LTTng has quite good documentation, but sometimes it’s not very clear, especially without deep knowledge of this tool. It took me almost a day to setup live streaming of the traces. One of the problems I had was related to this section of documentation: lttng.org/docs/v2.9/#doc-lttng-live - due to a bug in documentation this part:
lttng create --live my-session
should be like this:
lttng create my-session --live
There is a PR for this change: 2.7, 2.8, 2.9: put –live after session name.
Local streaming
Local means that traces are streamed to the same machine.
1. Target terminal session - create live session which triggers creation of relay daemon and enable kernel’s events:
$ lttng create my-session --live
Spawning a session daemon
Spawning a relayd daemon
Session my-session created.
Traces will be written in net://127.0.0.1
Live timer set to 1000000 usec
$ lttng enable-event --kernel --all
All Kernel events are enabled in channel channel0
2. Remote terminal session - list available tracing sessions and view it:
$ babeltrace --input-format=lttng-live net://localhost # list available tracing sessiong
net://localhost/host/laptop-name/my-session (timer = 1000000, 9 stream(s), 0 client(s) connected)
$ babeltrace --input-format=lttng-live net://localhost/host/laptop-name/my-session
[21:00:10.187779571] (+0.000000535) name sched_switch: { cpu_id = 2 }, ...
[21:00:10.187780930] (+0.000001359) name timer_hrtimer_cancel: { cpu_id...
[21:00:10.187782227] (+0.000001297) name timer_hrtimer_cancel: { cpu_id...
3. Target terminal session - start trace session:
$ lttng start
Tracing started for session my-session
Remote streaming
Remote means that traces are sent from one machine to another. For test purpose I created a headless Ubuntu Server VM which will “generate” traces, let’s name it guest-machine.
Note that on remote (receiver) machine ports 5342 and 5343 should be opened.
1. Target (guest-machine) - if --set-url
parameter is passed the relay daemon won’t start:
$ lttng create my-session --live --set-url=net://host-machine-ip
Spawning a session daemon
Session my-session created.
Traces will be written in net://host-machine-ip
Live timer set to 1000000 usec
2. Remote (host-machine) - start the daemon:
$ lttng-relayd
3. Target - enable kernel events, note that relay daemon should be started already on remote:
$ lttng enable-event --kernel --all
All Kernel events are enabled in channel channel0
$ lttng start
Tracing started for session my-session
4. Remote - view traces:
$ babeltrace --input-format=lttng-live net://localhost
net://localhost/host/guest-machine-ip/my-session (timer = 1000000, 2 stream(s), 0 client(s) connected)
$ babeltrace --input-format=lttng-live net://localhost/host/guest-machine-ip/my-session
[21:00:10.187779571] (+0.000000535) name sched_switch: { cpu_id = 2 }, ...
[21:00:10.187780930] (+0.000001359) name timer_hrtimer_cancel: { cpu_id...
[21:00:10.187782227] (+0.000001297) name timer_hrtimer_cancel: { cpu_id...