Objective: learn to use DTrace, in conjunction with the OpenGrok kernel source browser. We will add the kernel debugger "mdb -k" to the mix later in the course. A good slide deck to review DTrace features (I showed it in class): http://d3s.mff.cuni.cz/teaching/crash_dump_analysis/slides/10-dtrace.pdf [note that sun.com links are defunct; search for the page name, and you might find it on Oracle's port of the Sun blogs. Copies of some key documents, such as the DTrace User Guide, can be found in the course directory or in local/] In my experience, DTrace is best learned from examples. Start with kill.d, sshkeysnoop.d, and shellsnoop.d as suggested in http://www.brendangregg.com/dtrace.html: http://brendangregg.com/DTrace/kill.d http://brendangregg.com/DTrace/sshkeysnoop.d http://brendangregg.com/DTrace/shellsnoop.d See also ktrace.d (http://dtracebook.com/index.php/Kernel:ktrace.d) from the http://dtracebook.com/ collection. These examples illustrate how a follow the activities of a process from the kernel by intercepting its syscalls. Run these and make sure you understand all the details. (E.g.: what is copyinstr() and why is it necessary? What is the difference between 'self' and 'this', and why are both needed? Etc.) Aggregation is a major feature of DTrace, which helps profile applications' performance and the overall system activities. Study the 'DTrace one-liners' http://brendangregg.com/DTrace/dtrace_oneliners.txt for examples of aggregation, and run these on your system. Then study http://brendangregg.com/DTrace/bitesize.d --------------- Exercises: --------------- 1. When a process gets created, print the info about its parent and 'grandparent' processes, including their names and pids. (Note that since D does not include loops, printing its entire parent chain would be tricky. Loops were not included in D on purpose: DTrace designers did not want users to be able to accidentally create endless loops in the kernel.) 2. Trace the code path through which signals are delivered to processes (e.g., by 'kill'). Find out which parts of the proc_t process descriptor are used in the process, especially which 'struct proc *' members enable the delivery. Hint: combine the system call's function boundary tracing of ktrace.d with kill.d . Use pointer-chasing off the curthread 3. Notice that kill.d contains a (minor) bug. For example, suspending and unsuspending emacs results in root@openindiana:/home/sergey# ./kill.d FROM COMMAND SIG TO RESULT 940 bash 25 4294966305 0 991 emacs-gtk 24 4294966305 0 940 bash 25 4294966305 0 Clearly, 4294966305 is not a valid PID. What is going on and how to fix it? . . . . . . . Hint: read 'man -s2 kill' (the specification for the kill syscall) carefully. Hint: recall that DTrace variables' default type may not be the most natural type for a particular syscall argument (or return value).