Taking a full-system trace

[toddlipcon]

I noticed that when I trace an individual process or set of threads, there's a significant overhead added to context switching in the kernel as the context switch code needs to enable/disable intel_pt hooks. I'm trying to understand performance of a context-switch-heavy program, so this overhead is somewhat problematic.

I managed to collect a whole-system trace using something like:

sudo perf record -o /tmp/xdir/perf.data --event=intel_pt/cyc=1,cyc_thresh=1,mtc_period=0,noretcomp=1/u --timestamp    -C 8,9,10,11  -a sleep 0.001

which I can then decode with 'magic-trace decode'. But, it seems that if I try to also capture kernel events (with /uk instead of /u), things go south (I get weird call stacks where it looks like my functions never return).

Has anyone successfully done a whole-system trace (or a particular CPU trace, like I tried above)?

[cgaebel]

First of all, I'm sorry magic-trace didn't do the right thing. We tried to ship something that works, and it tends to only mostly work if you stick to the golden magic-trace attach path. Users like yourself get the joy of dealing with bugs in magic-trace, bugs in perf, bugs in IPT itself, and bugs in the integration of all three of those layers.

I haven't done any whole system-tracing with Intel PT. Perhaps unsurprisingly, I've mostly used it to trace single processes that monopolize a single isolated core.

Some shots in the dark:

• I'm a little surprised that perf decode didn't just work, though. Does perf record report any decode errors when it ran?
• Try is tracing a single core instead of 4.
• What version of perf are you running?
• Does magic-trace warn about missing kcore?
• Sometimes magic-trace creates odd stair-stepping traces when it is unable to close stack frames correctly, but the timeline still makes sense (modulo some spans that failed to close at the right time). If that happened to you, the trace may still be useful. It may also point you to the particular spans that magic-trace is failing to close, and that may help us debug what it's doing wrong. Known features magic-trace struggle with are self-modifying code (except in the kernel and when using a perf that supports kcore) and exceptions (in most languages). If it is a problem of us rendering something incorrectly, then if you give us a subset of your intel PT output that you think should create a reasonable trace (something like https://github.com/janestreet/magic-trace/blob/master/test/btree_rebalance_decode_error.perf) we can take a look.