Tag Archives: Endless

Heap profiling of gnome-software

The last week has been a fun process of starting to profile gnome-software with the aim of lowering its resource consumption and improving its startup speed. gnome-software is an important part of the desktop, so having it work speedily, especially on resource constrained computers, is important. This work is important for Endless OS, and is happening upstream.

To start with, I’ve looked at gnome-software’s use of heap memory, particularly during startup. While allocating lots of memory on the heap isn’t always a bad thing (caches are a good example of heap allocations being used to speed up a program overall), it’s often a sign of unnecessary work being done. Large heap allocations do take a few tens of milliseconds to be mapped through the allocator too. To do this profiling, I’ve been using valgrind’s massif tool, and massif-visualizer to explore the heap allocations. I could also have used heaptrack, or gobject-list, but they’re tools to explore another time.

Profile your app

Before diving into the process of optimising, the summary is that this work dropped gnome-software’s pixbuf heap usage by 24MB, and its non-pixbuf heap usage after initialisation (i.e. at the point when the main window is visible and ready to use) by 12%, from 15.7MB to 13.7MB (on my set of flatpak repositories on Fedora 32). I’ve been doing this work upstream, and it’ll trickle down to the downstream copy of gnome-software in Endless OS.

There is more low-hanging fruit to explore, and plenty of opportunities to dive in and trim more memory usage from gnome-software, or other apps. If you’re interested, please dive in! Get in touch if you have questions, or post them on GNOME’s Discourse instance and tag me. I’ll be happy to help!

How to profile heap usage

Profiling heap usage using massif is an iterative process: run your program under massif, do some actions in the program, quit, then open the resulting massif.out.pid file in massif-visualizer and see where allocations are coming from. Pick an allocation which looks large or unnecessary, find it in the code, optimise the code (if possible), and then repeat the process.

When running it, I wait for gnome-software to finish loading its main window, then I exit; so all this profiling work is for allocations during startup.

I run massif using this script, which I’ve put in ~/.local/bin/massif:

export G_SLICE=always-malloc
exec valgrind --tool=massif --num-callers=50 --suppressions=/usr/share/glib-2.0/valgrind/glib.supp --trace-children=no --threshold=0.1 --alloc-fn=g_malloc --alloc-fn=g_object_new --alloc-fn=g_malloc0 --alloc-fn=g_malloc0_n --alloc-fn=g_malloc_n --alloc-fn=g_realloc --alloc-fn=g_realloc_n --alloc-fn=g_slice_alloc --alloc-fn=g_slice_alloc0 --alloc-fn=g_type_create_instance --alloc-fn=g_object_new_internal --alloc-fn=g_object_new_with_properties --alloc-fn=g_object_newv --alloc-fn=g_object_new_valist --alloc-fn=g_try_malloc --alloc-fn=g_try_malloc_n --alloc-fn=g_hash_table_realloc_key_or_value_array --alloc-fn=realloc_arrays --alloc-fn=g_hash_table_resize --alloc-fn=g_hash_table_maybe_resize --alloc-fn=g_hash_table_insert_node --alloc-fn=g_hash_table_insert_internal --alloc-fn=g_hash_table_setup_storage "$@"

All the --alloc-fn arguments hide internal GLib functions so that the output is a little easier to interpret directly. There currently isn’t a way to store them in a config file or suppression file.

Some typical output from massif-visualizer before any code improvements:

massif-visualizer output from before code improvements to gnome-software, libxmlb or json-glib. The majority of the allocations are for pixbufs.

The window shows heap allocations against time in instructions executed. The breakdown of where each allocation came from is known in detail at key snapshots (which are expandable in the side pane), and the total heap usage is known in summary for the other snapshots, which allows the graph to be drawn. Allocations coming from different functions are coloured differently in the graph.

There are two sets of allocations to focus on: the red plateau between time 1e+10 and 1.6e+10, and the orange step from time 1e+10 onwards.

Selecting the red plateau shows the backtrace which led to its allocation in the side pane, and (despite some missing debug symbols, leading to the ‘???’ entries), it seems to have come from within libjpeg, as part of loading a JPEG pixbuf. gnome-software has various large JPEG images which are displayed in the featured app banners. It seems that libjpeg makes some big temporary allocations when loading a JPEG.

The orange step from 1e+10 onwards is another target. Looking at the backtraces, it seems it’s a series of similar allocations for pixbuf pixel storage for the featured app banners and for app icons. Some quick calculations show that each 1024×400 pixel banner will take around 6.5MB of memory to store its uncompressed pixels (at 16B per pixel).

From the graph and the backtraces, it seems that almost 100MB is used for pixbuf data for featured app banners. At 6.5MB per banner, that’s 15 banners, which seems reasonable. But actually gnome-software limits itself to 5 banners, so something’s amiss.

Style providers aren’t cheap

After adding some debug prints in GTK where it loads the pixbufs for CSS background properties, it became evident that the same few images were being loaded multiple times. CSS is used to style each featured app tile, including setting the background, since that allows a lot of artistic freedom quite easily. However, the CSS was being refreshed and set a few times for each tile, with a new GtkCssProvider each time. The old provider was staying in place, but with its properties overridden. This included the previously-loaded background image, which remained loaded but unused (essentially, leaked!). With that chased down, it was possible to fix the problem.

Back to profiling

With one issue investigated and fixed, the next step is to do another profiling run, find another target for reducing heap allocations, and repeat.

While we might have fixed one pixbuf bug in gnome-software, it does still use a lot of memory for pixbufs, since it displays a lot of high-resolution app icons. Those pixbuf allocations occupy a lot of space in the massif-visualizer view, and take up a large percentage of the ‘threshold’ of heap allocations which massif includes in its traces.

massif provides the --ignore-fn argument to allow certain allocations to be ignored, so that you can more easily profile others. So I did further profiling runs with a series of --ignore-fn arguments to ignore pixbuf allocations.

massif-visualizer output from before code improvements to gnome-software, libxmlb or json-glib, with pixbuf allocations ignored.

With the --ignore-fn arguments, and increasing the ‘Stacked diagrams’ level in the toolbar to show more individual areas on the graph, it’s now possible to see more detail on the largest non-pixbuf allocations, and hence easier to choose where to focus next.

massif-visualizer output from before code improvements to gnome-software, libxmlb or json-glib, with pixbuf allocations ignored, and more stacked diagrams shown.

From this screenshot, perhaps the next place to focus on would be GHashTable creation and insertions, since that totals around 1MB of the heap usage (once pixbufs are ignored).

Summary

I have iterated through the gnome-software massif profiles a few times, and have submitted various other fixes to gnome-software and libxmlb which are in the process of being reviewed and merged, but I won’t walk through each of them. There are still improvements to be made in future: gnome-software is quite complex!

In total, the changes reduced gnome-software’s heap usage at startup by 26MB, though the actual numbers will vary on other systems depending on how often feature tiles get refreshed, and how many apps and repositories you have configured.

These changes have not made a significant improvement to the startup time of gnome-software, which is more significantly influenced by network activity and file parsing (and the subject of some future work).

Hopefully this post gives a workable introduction to how to use massif on your own software. Please speak up if you have any questions. If you do profiling work on your software, please blog about it — it would be interesting to see what improvements are possible.

GUADEC 2018 thoughts

GUADEC this year was another good one; thank you to the organisers for putting on a great and welcoming conference, and to Endless for sending me.

Unfortunately I couldn’t make the first two days due to a prior commitment, but I arrived on the Sunday in time to give my talks. I’ve got a lot of catching up to do with the talks on Friday and Saturday — looking forward to seeing the recordings online!

The slides for my talk on the state of GLib are here and the notes are here (source for them is here). I think the talk went fairly well, although I imagine it was quite boring for most involved — I’m not sure how to make new APIs particularly interesting to listen to!

The slides for my talk on download management on metered connections (the ‘Mogwai’ project) are here and the notes are here (source for them is here). I think this talk also went fairly well, and I’m pleased by how many people turned up and asked insightful questions. As I said in the talk, my time to spend on this project is currently limited, but I am interested in mentoring new contributors on it. Get in touch if you’re interested.

During the birds of a feather days, I spent most of my time on GLib, clearing out old bugs. We had the GLib BoF during the GTK+ one on Monday. The notes are here. Emmanuele has already done a good writeup of the results of the BoF here; and Matthias has written up the GTK+ BoF itself here.

There were some good discussions over dinner during the BoF days about people’s niggles with GLib, which has set a few ideas in motion in my head which I will try and explore over the coming few months, once the 2.58 release is out of the way.

It was good to catch up with everyone, great to see Almería and sample its food and drink, and nice to finally meet some of my colleagues from Endless for the first time!

GTK+ hackfest and FOSDEM

Courtesy of my employer, Endless (we’re  hiring), I’m at the GTK+ hackfest in Brussels, which is acting as my warm up for FOSDEM 2018. I’m representing the assorted GLib maintainers, aiming to look at the roadmap for GLib 2.58, and what we need to do to finish off GLib 2.56. If you’ve got suggestions for new features or changes to GLib, get in touch or file a bug!

End of year thoughts

Inspired by others, I thought doing a retrospective on 2017 would be an interesting thing to look back on in a year’s time and see what’s changed.

Work things

December 2017 marked a year of me working for Endless. It’s been twelve months of fixing small bugs, maintaining some OS components, poking my nose into lower parts of the OS than I’m used to, and taking on one or two big projects. I spent a significant amount of time on a project to add new distribution features to libostree and flatpak. That’s something which will hopefully be rolling out in early 2018. It was good to be able to get fairly deeply involved with a new component at a lower level in the stack. More of that in 2018!

I also spent some of my time in 2017 picking up a bit more of the GLib maintenance workload. I’m not sure how much of a difference it’s made to the bug backlog, but it’s kept me occupied anyway.

Hobby things

For most of my working life, I’ve had the luxury of being able to work on FOSS software (mostly in the GNOME ecosystem) as my day job, and as a result, quite a few of my hobby projects are actually maintained during the day. The ones which aren’t have suffered during 2017, because time and energy are limited. I’ve been thinking of ways to ensure that code gets maintained, but haven’t come up with any good solutions in 2017. That’s one to carry over into 2018.

Trips

2017 was a bit less of a plane-heavy year than 2016, but some trips still happened:

  • FOSDEM, catching up with old friends and colleagues, and where the start of the current phase of GLib maintenance started.
  • A week of caving in South Wales, including a trip down the fantastic Dan-yr-Ogof cave (the short round), which included floating down an underground canal on an inflatable swimming pool ring.
  • A week of walking in the Glencoe area, where the weather was uncharacteristically cooperative, and the views were, predictably, pretty good.
  • A party in London to celebrate Endless’ 5th birthday. As always, it was good to spend quality time with my Endless colleagues in endless pubs.
  • Two weeks of caving in Austria, finding some new cave, and exploring further into existing cave. This is something I’m hoping to repeat in future.
  • GUADEC in Manchester, right on the back of the Austria trip (including some fun in posting a laptop to myself so I could have it at the conference). I gave a talk, which some people listened to. We also went on a walk in the Peak District, which was good fun (even if the weather was a bit grey).
  • Two weeks of long-distance trekking in the Svaneti region of Georgia. An excellent destination, with excellent cheese bread. We derived continual amusement from the guide’s dry humour, and the ‘helpful’ comments left by others on the trek information we were using. I did not get struck by lightning.
  • A long weekend in Stockholm to explore the city and catch up with friends. Stockholm has good running!

The outdoors

2017 has definitely been a year of taking advantage of living in the north of England.

  • Around 40 caving trips on weeknights and weekends, which have been interesting and (mostly) fun.
  • 12 fell races, a fun run along with a friend for part of their Bob Graham round, and my first ultra.
  • Running really took off for me: around 1300km run in total (and 57km of ascent), and about 150 hours of 2017 spent running.

Reading and listening

Gigs were a bit thin on the ground: despite there being plenty on in my local area, I always had something else to do. Despite that:

  • Insomnium were good, though I had to leave before the end because of trains.
  • Breabach were very good, and a band I hadn’t heard before going to the gig. Now a favourite.
  • Kreator sounded uncannily like their last live album, but were otherwise enjoyable.
  • Opeth were pretty fantastic, playing a good variety of new and old stuff.

I managed to read only 13 books in 2017, though that number is largely padded out by some short stories I read just to reach my yearly target. That’s not quite fair, though; I read 3250 pages in total. Most recommenable: Where Late the Sweet Birds Sang; most disappointing: Hiroshima.

GUADEC 2017: sun, rain, Coverity, walks

GUADEC 2017 has ended in Manchester. It’s been great; thanks to the organisers and sponsors for a fun conference (this year’s highlight: a preponderance of Tiki bars).

We’ve had sun and heat, and we’ve had rain and more rain. Often within the same hour. On the final day of the conference, a group of us went out to Edale to do some walks to see the Peak District, a national park area near Manchester. This is an area I’ve visited many times before, so it was fun to be able to show it to GNOME people.

This year I gave a talk about the Coverity scans I’ve been running on various GNOME and freedesktop modules for the last year. The slides are online and the video will be up with the rest of the GUADEC videos. If you have a security-critical (or other) module which you’d like to be included in the scan set, let me know. Coverity’s good at finding bugs in complex control flows, but you do need to put some time into triaging its reports. I’m happy to provide guidance about using it.

I spent a fair amount of time during the unconference days reviewing Simon McVittie’s D-Bus work to add support for app-containers into the D-Bus specification and dbus-daemon. This is the first part of an effort to improve support for exposing unconfined D-Bus services to confined app-containers safely and efficiently. The rest of my time was spent working on exciting support for updating flatpak over the LAN for Endless OS. I’ll blog about this more in future.

Thanks to the GUADEC team for organising a great conference, the conference sponsors, and to my employer, Endless, for sponsoring me to go.