Looking at Illumos (OpenSolaris) process descriptors with MDB. MDB is the Solaris' "modular debugger". It has a different command structure than GDB, and it takes practice and exploration to get used to. However, MDB offers very rich commands for looking at the internals of a running kernel ("mdb -k", started as root). Here is my session. My comments start with SB: sergey@openindiana:~$ sudo mdb -k Password: Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp scsi_vhci zfs sata sd ip hook neti sockfs arp usba stmf stmf_sbd fctl lofs random idm cpc fcip crypto sppp nfs ufs logindmux ptm ipc ] > ::help Each debugger command in mdb is structured as follows: [ address [, count]] verb [ arguments ... ] ^ ^ ^ ^ the start --+ | | +-- arguments are strings which can be address can be an | | quoted using "" or '' or expression | | expressions enclosed in $[ ] | | the repeat count --+ +--------- the verb is a name which begins is also an expression with either $, :, or ::. it can also be a format specifier (/ \ ? or =) For information on debugger commands (dcmds) and walkers, type: ::help cmdname ... for more detailed information on a command ::dcmds ... for a list of dcmds and their descriptions ::walkers ... for a list of walkers and their descriptions ::dmods -l ... for a list of modules and their dcmds and walkers ::formats ... for a list of format characters for / \ ? and = For information on command-line options, type: $ mdb -? ... in your shell for a complete list of options > ::dcmds $< - replace input with macro $<< - source macro $> - log session to a file $? - print status and registers $C - print stack backtrace $G - enable/disable C++ demangling support $M - list macro aliases SB: ::ps is a typical walker command. It walks the process table and prints a summary line per process. These summaries helpfully include the address of each proc_t structure (process descriptor): > ::ps S PID PPID PGID SID UID FLAGS ADDR NAME R 0 0 0 0 0 0x00000001 fffffffffbc2e330 sched R 3 0 0 0 0 0x00020001 ffffff014e183028 fsflush R 2 0 0 0 0 0x00020001 ffffff014e184020 pageout R 1 0 0 0 0 0x4a004000 ffffff014e08f018 init R 4208 1 4207 4207 0 0x42000000 ffffff0160825060 sshd R 4195 1 4178 4178 0 0x4a004900 ffffff01609db0b8 modunload R 4176 1 4159 4159 0 0x4a004900 ffffff01609d20d0 modunload R 3981 1 3886 3886 0 0x4a004900 ffffff01606a8048 modunload R 4153 1 4153 4153 0 0x52010000 ffffff015488f010 sendmail R 4146 1 4146 4146 25 0x52010000 ffffff01609d70c0 sendmail R 4142 1 4142 4142 0 0x42000000 ffffff0160828058 nscd R 4128 1 4128 4128 0 0x42000000 ffffff01546b10b0 automountd R 4129 4128 4128 4128 0 0x42000000 ffffff0153902008 automountd R 1441 1 1440 1440 101 0x4a004000 ffffff015ff99028 gam_server R 1422 1 1165 1165 101 0x4a014000 ffffff015b4c2018 firefox R 1435 1422 1165 1165 101 0x4a014000 ffffff015ae0a008 run-mozilla.sh R 1439 1435 1165 1165 101 0x4a004000 ffffff015b4bd020 firefox-bin R 1270 1 1165 1165 101 0x4a004000 ffffff01598270b8 gnome-terminal R 4156 1270 4156 4156 101 0x4a014000 ffffff0160818078 bash SB: Let's look at one proc_t, pick the last line above. Printing memory at ffffff0160818078 (it's 64 bit system) interpreting its contents as a struct proc_t: > ffffff0160818078::print proc_t { p_exec = 0xffffff0158c36c80 p_as = 0xffffff016054f940 p_lockp = 0xffffff014ab878c0 p_crlock = { _opaque = [ 0 ] } p_cred = 0xffffff0156ba4810 p_swapcnt = 0 p_stat = '\002' p_wcode = '\0' p_pidflag = 0 p_wdata = 0 p_ppid = 0x4f6 p_link = 0 p_parent = 0xffffff01598270b8 p_child = 0xffffff01609de0b0 p_sibling = 0xffffff0160817080 p_psibling = 0 p_sibling_ns = 0 p_child_ns = 0 p_next = 0xffffff015488f010 p_prev = 0xffffff01609d20d0 u_start = { tv_sec = 2013 Jan 18 19:35:35 tv_nsec = 0x29b0b934 } u_ticks = 0x1b08bb1 u_comm = [ "bash" ] u_psargs = [ "bash" ] u_argc = 0x1 u_argv = 0x8047abc u_envp = 0x8047ac4 u_cdir = 0xffffff0158eee800 u_rdir = 0 u_mem = 0x1463 u_mem_max = 0x17aa SB: The PID of the process sits in a small 'stuct pid', pointed to from proc_t. "! grep pid" will help us filter relevant lines: > ffffff0160818078::print proc_t !grep pid p_pidflag = 0 p_ppid = 0x4f6 p_pidp = 0xffffff0162365030 p_lwpid = 0x1 p_t1_lgrpid = 0 p_tr_lgrpid = 0xffffffff p_ancpid = 0x4f6 SB: Now to print that struct: > 0xffffff0162365030::print 'struct pid' { pid_prinactive = 0 pid_pgorphaned = 0x1 pid_padding = 0 pid_prslot = 0x63 pid_id = 0x103c pid_pglink = 0xffffff0160818078 pid_pgtail = 0xffffff0160818078 pid_link = 0 pid_ref = 0x4 } SB: So the PID is 0x103c. Let's cross-check that with regular "ps". Luckily, we need not even leave the MDB shell: to run Unix shell commands from it we just need to prefix them with '!' > !printf "%d\n" 0x103c 4156 > !ps ax | grep bash 1422 ? S 0:00 /bin/bash /usr/bin/firefox 1435 ? S 0:00 /bin/bash /usr/lib/firefox/run-mozilla.sh /usr/lib/fire 1273 pts/1 S 0:00 bash 4212 pts/1 R 0:00 bash -c ps ax | grep bash 3595 pts/2 S 0:00 bash 3598 pts/2 S 0:00 sudo bash 3599 pts/2 S 0:00 bash 4156 pts/3 S 0:00 bash SB: Indeed, the last shell it is. SB: Note that we can follow any pointers and print memory that they point to, as long as we know the type of the values or structs contained there. We can also print type definitions. Option '-a' prints offsets of each member of a struct type: > ::print -a kthread_t 0 { 0 t_link 8 t_stk 10 t_startpc 18 t_bound_cpu 20 t_affinitycnt 22 t_bind_cpu 24 t_flag 26 t_proc_flag 28 t_schedflag 2a t_preempt 2b t_preempt_lk 2c t_state 30 t_pri 32 t_epri 34 t_cpri 36 t_writer 37 t_bindflag 38 t_pcb { 38 val } 170 t_forw 178 t_back 180 t_thlink 188 t_lwp 190 t_procp SB: That last one is a pointer to the proc_t that a process belongs to. SB: =K can be used to get the value of a symbol, i.e., the address that this symbol is known to reside at. (Why 'K'? See ::formats . Addresses are 8 bytes long. For a 32 bit system we'd use 'X', which on this 64bit machine just gives us the lower 4 bytes of the address, not very useful) > getpid=K fffffffffba44168 > getpid=X fba44168 SB: Disassemble kernel code at that address: > getpid::dis getpid: pushq %rbp getpid+1: movq %rsp,%rbp getpid+4: subq $0x10,%rsp SB: This is how curthread from /uts/common/sys/thread.h is inlined: (the assembly implementation of threadp() is in /uts/intel/asm/thread.h, specific to Intel architectures; there are others for Sparc, etc.) getpid+8: movq %gs:0x18,%rax SB: And this is the first line of getpid() from /uts/common/syscall/getpid.c, p = ttoproc(curthread), and ttoproc(x) is macro-defined as ((x)->t_procp) in thread.h. Note that 0x190 is exactly the offset of t_procp in kthread_t. Convince yourself that the other offsets in this chain of pointer derefences (C's '->' operator) make sense. Look at the code of getpid() if you get confused by assembly logic, but don't worry about the "ZONE" branch yet. getpid+0x11: movq 0x190(%rax),%r9 getpid+0x18: movq 0xb0(%r9),%r8 getpid+0x1f: movl 0x4(%r8),%eax getpid+0x23: movl %eax,-0x8(%rbp) getpid+0x26: testl $0x400,0xcc(%r9) getpid+0x31: jne +0x9 getpid+0x33: movl 0x34(%r9),%eax getpid+0x37: movl %eax,-0x4(%rbp) getpid+0x3a: jmp +0x2c getpid+0x3c: movq %gs:0x18,%rax getpid+0x45: movq 0x190(%rax),%r8 getpid+0x4c: movq 0x620(%r8),%r8 getpid+0x53: movq 0x150(%r8),%r8 getpid+0x5a: movq 0xb0(%r8),%r8 getpid+0x61: movl 0x4(%r8),%eax getpid+0x65: movl %eax,-0x4(%rbp) getpid+0x68: movq -0x8(%rbp),%rax getpid+0x6c: leave getpid+0x6d: ret SB: Raw address would work, too. Note that the disassembler looks up the symbol at that address, hits "getpid", and uses that for the listing. Symbol lookup can be done with the "::nm" command, too. > fffffffffba44168::dis getpid: pushq %rbp getpid+1: movq %rsp,%rbp getpid+4: subq $0x10,%rsp getpid+8: movq %gs:0x18,%rax getpid+0x11: movq 0x190(%rax),%r9 getpid+0x18: movq 0xb0(%r9),%r8 > fffffffffba44168::nm Value Size Type Bind Other Shndx Name 0xfffffffffba44168|0x000000000000006e|FUNC |GLOB |0x0 |3 |getpid SB: So you can navigate the process table, in particular visit the proc_t descriptors of parent processes, child processes, etc. Some other special commands that interpret memory are on page 156 of the textbook (Section 2.13). > ffffff0160818078::pmap SEG BASE SIZE RES PATH ffffff0154899820 0000000008044000 16k 16k [ anon ] ffffff04b96a0938 0000000008050000 740k 724k /usr/bin/bash ffffff04b96a08d8 0000000008118000 44k 44k /usr/bin/bash ffffff04b96a0998 0000000008123000 220k 196k [ anon ] ffffff0154648d78 00000000fe7d0000 56k 56k /lib/libsocket.so.1 ffffff0162342af8 00000000fe7ee000 4k 4k /lib/libsocket.so.1 ffffff0161167898 00000000feaf0000 456k 416k /lib/libnsl.so.1 ffffff04b96a0c98 00000000feb72000 8k 8k /lib/libnsl.so.1 ffffff015ffca690 00000000feb74000 20k 12k [ anon ] ffffff015ffa0ea0 00000000fecf0000 64k 16k [ anon ] ffffff016233e140 00000000fed03000 4k 4k [ anon ] ffffff0162350f00 00000000fed0c000 76k 32k /usr/lib/locale/en_US.UTF-8/ ffffff01611672f8 00000000fed20000 24k 12k [ anon ] ffffff015502dd40 00000000fed30000 4k 4k [ anon ] ffffff04b9738c48 00000000fed40000 180k 140k /lib/libcurses.so.1 ffffff0162342978 00000000fed7d000 28k 28k /lib/libcurses.so.1 ffffff015ff9d8a8 00000000fed84000 8k 8k [ anon ] ffffff015ff9dde8 00000000fed90000 1216k 940k /usr/lib/libc/libc_hwcap1.so ffffff015ff85e08 00000000feec0000 36k 36k /usr/lib/libc/libc_hwcap1.so ffffff04b9738d68 00000000feec9000 8k 8k [ anon ] ffffff04b973b340 00000000feed0000 24k 24k /lib/libgen.so.1 > ffffff0160818078::pfiles FD TYPE VNODE INFO 0 CHR ffffff01608c8040 /dev/pts/3 1 CHR ffffff01608c8040 /dev/pts/3 2 CHR ffffff01608c8040 /dev/pts/3 3 DOOR ffffff04b96a5640 /var/run/name_service_door [door to 'nscd' (proc= ffffff0160828058)] 255 CHR ffffff01608c8040 /dev/pts/3 > ::ptree fffffffffbc2e330 sched ffffff014e183028 fsflush ffffff014e184020 pageout ffffff014e08f018 init ffffff0160825060 sshd ffffff01609db0b8 modunload ffffff01609d20d0 modunload ffffff01606a8048 modunload ffffff015488f010 sendmail ffffff01609d70c0 sendmail ffffff0160828058 nscd ffffff01546b10b0 automountd ffffff0153902008 automountd ffffff015ff99028 gam_server ffffff015b4c2018 firefox ffffff015ae0a008 run-mozilla.sh ffffff015b4bd020 firefox-bin ffffff01598270b8 gnome-terminal ffffff0160818078 bash ffffff01609de0b0 emacs-gtk ffffff0160817080 bash <<<<<<<< Look here, our shell ffffff016054d038 sudo ffffff016054b040 bash Now you can explore further up and down the tree and around the textbook's Chapter 2.