Comments on: Reference count debugging with systemtap

By: iain

iain — Thu, 15 Jul 2010 00:16:25 +0000

I did something similar by writing a valgrind plugin. The downside was that it was quite verbose, but with some filtering options it worked quite nicely. I meant to write a logfile analyser, but never got round to it and then found something else to do.

By: Eero Tamminen

Eero Tamminen — Wed, 14 Jul 2010 20:18:59 +0000

"The ability to set a maximum number of frames to print out in a backtrace would be useful, since I’ve found that I usually only need the first eight-or-so frames for this kind of work; any more frames just get in the way."

Fewer frames will also be faster, unwinding a lot of calls can get slow...

By: Philip Withnall

Philip Withnall — Wed, 14 Jul 2010 08:32:34 +0000

For debugging reference count problems, usually just the function name is what I need. If systemtap can't figure out the function name, the module name would definitely be useful. Other than that, a line number would be the most useful thing, although I don't know how possible that is.

The ability to set a maximum number of frames to print out in a backtrace would be useful, since I've found that I usually only need the first eight-or-so frames for this kind of work; any more frames just get in the way. It would be nice if the memory limit for each probe was raised as well (I guess there might currently be an option for it, but I haven't looked into it), since I ran into that when I was testing this out last night — seems that a particularly long backtrace made systemtap unhappy.

Thanks!

By: Mark Wielaard

Mark Wielaard — Wed, 14 Jul 2010 08:01:14 +0000

Very cool! So with the soon to be released SystemTap 1.3 you get two improvements that should make your life easier: - A new --ldd switch, which directs stap to slurp in all libraries that a binary links against, so you can drop the large -d -d -d list and just do: stap --ldd -d "${bindir}/empathy" refs.stp -c "${bindir}/empathy" - Some backtrace as string functions (sprint_[u]backtrace() and sprint_[u]stack()) which make the above trick Frank posted above even simpler (no need to convert by hand from hex addresses to symbols/modules, etc). I see you are using the print_ubacktrace_brief() variant instead of the print_ubacktrace() variant. Personally I thought the brief variant was a mistake since it was "too brief" and we needed something in between. What kinds of information do you want/need in a backtrace? address, module name, symbol name, offsets, function size, ...? We might be able to provide something even more flexible. The internal systemtap _stp_addr_sprint() function can already provide all or nothing, prefixes, suffixes, etc. But this isn't exposed to the script level (and I admit I don't yet know how to do that naturally, you quickly end up in some printf like syntax which might be overkill).

By: Philip Withnall

Philip Withnall — Wed, 14 Jul 2010 07:28:41 +0000

I'll have to try that out today, thanks.

Post updated re. the kernel-debuginfo package.

By: Frank Ch. Eigler

Frank Ch. Eigler — Wed, 14 Jul 2010 00:54:34 +0000

How to indent based upon the nearby refcount variable? For the header lines, printf ("%-*s %s %p ref (%u)\n", refcount, "", type, object, refcount) (see the %-*s part). For indenting the print_*backtrace() ... not quite so easy, since each line would need to be manipulated. But something like this could work: bt = ubacktrace() # results in a string of hex addresses addr = tokenize(bt, " ") while (addr != "") { printf("%-*s %s\n", refcount, usymdata(strtol(addr,16))) addr = tokenize ("", " ") } which could be plopped into a separate function. BTW, if you're not probing the kernel itself, you won't need the kernel-debuginfo package.