Historical reference: https://en.wikipedia.org/wiki/Terminus_%28god%29

-------------------[ SELinux and LSM ]-------------------

SELinux is implemented on top of LSM, as a stackable module.  LSM
provides a collection of hooks distributed throughout the kernel code
paths that have to do with file accesses and other security-sensitive
operations; modules register their callback functions for these hooks
(see http://src.illumos.org/source/s?refs=security_add_hooks&project=linux-master
for the calls to the security_add_hooks() that registers sets of callbacks),
and these functions are called in order when a hook is traversed.

The macro that traverses a hook list (note the style of the list
traversal and the varargs macro):
http://src.illumos.org/source/xref/linux-master/security/security.c#call_int_hook

Typical use of the macro:
http://src.illumos.org/source/xref/linux-master/security/security.c#880

Note that the prototypes of these functions are defined in
http://src.illumos.org/source/xref/linux-master/include/linux/security.h.
You can look at the positioning of these security_* hooks in the kernel.
The following paper analyzed this positioning to demonstrate 
the property of _complete mediation_ (all relevant paths have 
a hook; there are no unmonitored ones):
"Using CQUAL for Static Analysis of Authorization Hook Placement":
http://www.cse.psu.edu/~trj1/papers/usenix_sec2002.pdf


The types of the callback functions must match the respective definitions:
http://src.illumos.org/source/xref/linux-master/include/linux/lsm_hooks.h#security_list_options
Note the huge union of function pointer types; static type checking should 
catch a hook that does not comply with the wrong type of argument(s).

For the original paper on LSM:
http://www.kroah.com/linux/talks/ols_2002_lsm_paper/lsm.pdf 

SELinux hooks start with selinux_* and are be registered with LSM at 
http://src.illumos.org/source/xref/linux-master/security/selinux/hooks.c#6352
in selinux_init().

Follow one the selinux_* functions in
http://src.illumos.org/source/xref/linux-master/security/selinux/hooks.c
as it extracts the security labels from inode and task_struct objects,
and feeds them into a bit vector matching routine (avc_*, for access
vector cache) routine.  A compiled and loaded SELinux policy will be
represented as a collection of bit vectors to match; avc also provides
a caching layer. Some notes on the internals of AVCs:
https://www.imperialviolet.org/2009/07/14/selinux.html

-------------------[ Anomaly detection ]-------------------

SELinux is a "bag of permissions" type of policy: once permissions for
particular system calls and combinations of labels are granted, the
process may perform these operations any number of times, in any
order. 

Various other policy schemes enforce the order in which operations
happen. For example, _privilege separation_ has become a standard
for Internet-facing daemon processes:
      http://www.citi.umich.edu/u/provos/ssh/privsep.html

The correct sequences of operations can also be learned rather
that specified. Here is the paper that started system-call based anomaly 
detection in Unix: 
http://wenke.gtisc.gatech.edu/ids-readings/unix_process_self.pdf 
(there were many others since!).

For some discussion, see also
  http://cr.yp.to/unix/disablenetwork.html
(DJ Bernstein, a famous cryptographer, also wrote his own e-mail
 and DNS servers, with notable security records. See, e.g.,
 https://cr.yp.to/qmail/qmailsec-20071101.pdf ) 

-------------------[ Android SELinux ]-------------------

SELinux policies on a desktop or server systems tend to get quite large,
and are hard to debug (in Ubuntu or Debian, you can get the source of
the default SELinux policy in the package selinux-policy-source). 

Android, on the other hand, is making active use of SELinux:
https://source.android.com/security/selinux/

You can read https://www.internetsociety.org/sites/default/files/02_4.pdf,
or look at the slide deck
https://events.linuxfoundation.org/sites/events/files/slides/ abs2014_seforandroid_smalley.pdf

For a guide in reading the Android policy, follow
https://ge0n0sis.github.io/posts/2015/12/exploring-androids-selinux-kernel-policy/
(but use "git clone https://android.googlesource.com/platform/system/sepolicy" to
get the policy; its location in the Android tree has moved.)

This large free book 
http://freecomputerbooks.com/books/The_SELinux_Notebook-4th_Edition.pdf
can serve as a reference if you decide to do a SELinux-based project.

-------------------[ Rootkits ]-------------------

We looked at some examples in http://www.cs.dartmouth.edu/~sergey/cs258/rootkits/ 
today:  knark.c and knark.h, and its accompanying utility rootme.c ,
and http://phrack.org/issues/59/5.html (look in the stories/ subdirectory for code).

Adore and adore-ng are later (but still early Linux) kernel rootkits.
See http://phrack.org/issues/61/10.html and http://phrack.org/issues/61/14.html
Since these early LKM rootkits, Linux kernel module interfaces and loading
process has changed, adding "version magic" strings to discourage the distribution
of binary modules (and encourage vendors to ship drivers with source code).
http://phrack.org/issues/68/11.html has details on these changes and the new
process for building and loading an LKM.

It is hard to distinguish rootkits from DRM software, and, sometimes, from AV,
as both seek out and hook points in the target software where their functionality
can be added on certain code paths with least effort and most reliability. 
Two cases in point: 
 - the Sony rootkit of 2005, which impaired customer security:
     https://www.schneier.com/blog/archives/2005/11/sonys_drm_rootk.html
 - Kaspersky and McAfee products in 2006: 
     http://www.uninformed.org/?v=4&a=4&t=pdf