CEG 429/629:
Internet Security


College of Engineering & CS
Wright State University
Dayton, Ohio 45435-0001

Hardening an
Operating System

 

Prabhaker Mateti

 
Abstract:  We distinguish hardening a system from proper configuration and fortification.  In this lecture, we describe areas of tightening security during the design and construction of systems rather than after their deployment.
 
This work is supported in part by NSF DUE-9951380.
  10/28/03

Table of Contents

  1. Educational Objectives
  2. Hardening an Operating System
    1. Careful Recompilation
    2. Patching a Kernel
  3. Lab Experiment
  4. Acknowledgements
  5. References

Educational Objectives

  1. Understand hardening in contrast to proper configuration and fortification of a system.
  2. Become familiar with what makes and OS trusted and/or secure.

Hardening an Operating System

We distinguish, even though it is not common yet, hardening a system from proper configuration and fortification.  In this lecture, we describe areas of tightening security during the design and construction of systems rather than after their deployment. 

Careful Recompilation

Stack smashing (buffer overflow) attacks are among the most common.   By and large, these are programming errors that can be caught by analytical techniques.  Newer compilers are mechanizing these techniques.  For example, such vulnerabilities are carefully checked by the Immunix's "StackGuard" enhancements to the GNU egcs compiler.  An entire RedHat distribution has been recompiled with this compiler resulting in the distribution of Immunix.

Patching a Kernel

Patches are often released in binary form, but they they can also be at the source code level.  These replace section(s) of code in the kernel.  Often a patch is in response to a newly discovered security hole.  There are proactive modifications also in the open source OS.  For example, LIDS (Linux Intrusion Detection System)  is a series of kernel patches that enable loadable module and mount point locking  [www.soaring-bird.com.cn/oss_proj/lids/].

Several groups now release patches for the stock Linux kernel.  A group called "grsecurity" has gathered together kernel modifications suggested by many individuals and releases them as patches applied at the source code level using the standard system tool called patch.  Below is a list of features as claimed at http://www.grsecurity.net/

  1. Role-Based Access Control: User, group, and special roles; Role transition tables; IP-based roles; Non-root access to special roles; Special roles that require no authentication; Nested subjects; Variable support in configuration; And, or, and difference set operations on variables in configuration; Object mode that controls the creation of setuid and setgid files; Create and delete object modes; /dev/grsec entry for kernel authentication and learning logs; Next-generation code that produces least-privilege policies for the entire system with no configuration; Enhanced implementation of Trusted Path Execution; Full pathnames for offending process and parent process; RBAC status function for gradm; /proc/<pid>/ipaddr gives the remote address of the person who started a given process;
  2. Process-based Mandatory Access Control; Secure policy enforcement; Supports read, write, append, execute, view, and read-only ptrace object permissions; Supports hide, protect, and override subject flags; Supports the PaX flags; Shared memory protection feature; Integrated local attack response on all alerts; Subject flag that ensures a process can never execute trojaned code; Intelligent learning mode that produces least-privilege ACLs with no configuration; Full-featured fine-grained auditing; Resource ACLs; Socket ACLs; File/process ACLs; Capabilities; Protection against exploit bruteforcing; /proc/pid filedescriptor/memory protection; ACLs can be placed on non-existent files/processes; ACL regeneration on subjects and objects; Administrative mode to use for regular sysadmin tasks; ACL system is resealed up admin logout; Globbing support on ACL objects; Configurable log suppression; Configurable process accounting; Human-readable configuration; Not filesystem dependent; Not architecture dependent; Scales well: supports as many ACLs as memory can handle; No runtime memory allocation; SMP safe; Include directive for specifying additional ACLs; Enable, disable, reload capabilities; Userspace option to test permissions on an ACL; Option to hide kernel processes;
  3. Chroot restrictions: No attaching shared memory outside of chroot; No kill outside of chroot; No ptrace outside of chroot (architecture independent); No capget outside of chroot; No setpgid outside of chroot; No getpgid outside of chroot; No getsid outside of chroot; No sending of signals by fcntl outside of chroot; No viewing of any process outside of chroot, even if /proc is mounted; No mounting or remounting; No pivot_root; No double chroot; No fchdir out of chroot; Enforced chdir("/") upon chroot; No (f)chmod +s; No mknod; No sysctl writes; No raising of scheduler priority; No connecting to abstract unix domain sockets outside of chroot; Removal of harmful privileges via capabilities; Exec logging within chroot;
  4. Address space modification protection PaX: Page-based implementation of non-executable user pages; PaX: Segmentation-based implementation of non-executable user pages for i386 with negligible performance hit; PaX: Segmentation-based implementation of non-executable KERNEL pages; PaX: Mprotect restrictions prevent new code from entering a task; PaX: Randomization of stack and mmap base; PaX: Randomization of heap base; PaX: Randomization of executable base; PaX: Randomization of kernel stack; PaX: Automatically emulate sigreturn trampolines; PaX: No ELF .text relocations; No kernel modification via /dev/mem, /dev/kmem, or /dev/port; Option to disable use of raw I/O; Removal of addresses from /proc/<pid>/ [maps|stat];
  5. Auditing features Option to specify single group to audit; Exec logging with arguments; Denied resource logging; Chdir logging; Mount and unmount logging; IPC creation/removal logging; Signal logging; Failed fork logging; Time change logging;
  6. Randomization features Larger entropy pools; Randomized TCP Initial Sequence Numbers; Randomized PIDs; Randomized IP IDs; Randomized TCP source ports; Randomized RPC XIDs;
  7. Other features /proc restrictions that don't leak information about process owners; Symlink/hardlink restrictions to prevent /tmp races; FIFO restrictions; Dmesg(8) restriction; GID-based socket restrictions; Nearly all options are sysctl-tunable, with a locking mechanism; All alerts and audits support a feature that logs the IP of the attacker with the log; Stream connections across unix domain sockets carry the attacker's IP with them; Detection of local connections: copies attacker's IP to the other task; Low, Medium, High, and Custom security levels; Tunable flood-time and burst for logging.

Secure Operating Systems

There are "secure operating systems" and "trusted operating systems."  Alas, no one can offer technically rigorous definitions for these terms.  For example, we found the following quote, "Windows NT is a secure operating system but only if it's configured correctly." Nevertheless, operating systems that lay claim to either being secure or trusted are better designed and engineered from their inception with a concern for security.  Both NT,  several of Unix derivatives claim to be secure and trusted.

In a later lecture we will discuss the so-called Orange Book, and its classification of levels of security.

Trusted Operating Systems

The following is from a white paper by a company that sells a trusted operating system [www.argus-systems.com/]  It offers concrete enough details that we can ignore the specific slant it obviously has towards a single product.

Least Privilege

On any computer system, certain system programs or utilities must be granted the ability to bypass the security constraints normally imposed by the system. For example, in order to create a backup of all files on the system disks, an administrator must be able to run a backup program that is able to read all files on the disk, even though the administrator would not normally be allowed such access. Other powerful programs must also be carefully controlled, such as the programs to shut down the system, create new users, and repair damaged file systems. On a standard Unix system, the operating system has been designed so that one user ID, called root or superuser, can bypass all security restrictions and limitations. Windows NT systems exhibit similar vulnerabilities with the "System" and "Administrator" accounts.

The inability of standard Unix to grant only limited rights to a program is not the system's only weakness. In Unix, when one program starts another the newly created program runs with the user ID of the first program. This means that a malicious user who can exploit a bug in a root program may be able to start up an interactive root session. If a user is running as root, every program he runs will have unlimited privileges on the system.

On a trusted system the super user privilege is broken down into many smaller privileges. Thus the backup program may be able to read any file, but it cannot shut down the system, modify files, or send random network packets. The use of many limited capabilities instead of a single all-powerful mechanism is called the least privilege principle. On a trusted operating system, programs, not users, are given privileges. The backup program will have only the privileges it needs to do its job, and any program that it starts will not inherit the backup program's privileges. A specific user may have the right to run certain programs, but the user's session is in no way privileged. The programs with privileges are few, have limited power, and are carefully controlled.

The use of least privilege eliminates most of the security problems commonly reported with standard operating systems. Even if a specific super user bug may go unfixed, on a trusted operating system exploitation of the bug will not allow a malicious user to bypass all system security.

Labeling and Partitioning

The traditional Unix model allows all users to share all system resources, with access to file system objects limited by the owner of the object. This model, which is sufficient for systems requiring low levels of security, is unacceptable for sites that require enhanced levels of security or that cannot allow system security enforcement to be dependent on the user's voluntary adherence to site security policy.

In order to create a mechanism allowing users and processes to operate in isolation, it is necessary to add new security attributes to file system objects, processes, network interfaces, host addresses, and other system resources. These new attributes are called sensitivity labels (SLs).  The sensitivity labels can be used to restrict a process or user to either read-only access to an object or no access at all. Since SLs can not be modified by users or unprivileged processes, a system's security policy can be enforced without relying on user compliance. Access control using system-enforced labels is called mandatory access control (MAC).

Labeling and MAC can be applied at the network interface level. Incoming packets from a network interface are assigned SLs, which may be based on the source IP address or the network interface. Outgoing packets will have the label of the process or daemon that created them. An incoming or outgoing packet will be dropped if the SL of the packet is not valid for both the interface and the remote host. In addition, the SL can be inserted into the packet header so that trusted operating systems can share security information over the network.

Processes, files, and other resources that have the same label are said to be in the same compartment or partition. Programs, data, and network interfaces can be split into separate, isolated partitions with restricted access between them. Packets associated with one compartment cannot be read or spoofed by applications or network services in another.

Partitioning is a critical security feature when a system will be handling different classes of users and jobs, such as users from different departments or network daemons interacting with different interfaces or hosts. The ability to partition a network server is a key component of PitBull .comPack in providing the level of assurance needed to support critical network and transaction servers.

For example, network services, such as HTTP daemons and associated CGI scripts, can be given read-only access to some files (for example, critical web pages and CGI directories) and can be completely isolated from other resources (such as internal network interfaces, system files, and other network daemons). A network server configured in this way can protect its web pages and CGI scripts from ever being modified from an external connection, even if damaging bugs exist in the commercial software that allow incoming users to execute any random sequence of machine instructions. Access by the network services to other network interfaces and hosts can be easily limited as well.

Login Enhancements

Trusted operating systems employ enhanced account security features, including:

Authorizations and Privileges

Authorization is an attribute of a user account under a trusted operating system that enables the user to execute a subset of operating system applications and utilities. Privileges are attributes assigned to applications that give them different grades of access to processes or resources. Together, these two sets of attributes enable programs to behave differently toward different users. Applications can be extended to define and check new authorizations, providing a standardized protocol by which applications and the operating platform can communicate to grant permissions to users. Authorizations can be used to divide administrative duties into separate roles, which can then be assigned to different users.

Auditing

Trusted operating systems employ an enhanced auditing mechanism, which allows applications to trace a wider variety of security-related events than standard operating systems. The information on these transactions is appended to an audit trail in an isolated partition, which is protected by both discretionary and mandatory access control mechanisms. This approach prevents intruders from covering their tracks and eliminating traces of penetration attempts. By protecting the audit record, sufficient evidence is maintained to support litigation and law enforcement actions.

 

Independent Validation

Most IS and corporate managers, already hard-pressed to maintain daily systems operations, face significant barriers to incorporating new technologies and adequate systems security. Managers seeking to upgrade security on their systems are thus often forced to rely on vendor claims of security performance. As new software emerges and inevitable upgrades to existing software pour in, IS professionals typically assume that the vendors have a vested interest in the security of their products. Given the potential implications of security system failure, it is critical that managers concentrate on security solutions that have undergone independent evaluation, testing, and certification.

Trusted operating systems undergo evaluation of their overall design, verification of the integrity and reliability of their source code, and systematic, independent penetration evaluation. One of the most highly respected evaluation tools is the Information Technology Security Evaluation Criteria (ITSEC), an internationally recognized set of standards for the evaluation, testing, and certification of IT security products. ITSEC certification, performed by an independent body, provides the end user with confidence that the claims made about the security functionality of a product are valid. In addition, this certification indicates that these claims have been tested against a predetermined level of assurance and that the vendor has a high level of expertise and a strong commitment to security.


Lab Experiment

You may wish to do this experiment on your own home PC.

  1. Download the latest bundle of source code of Linux from www.kernel.org.
  2. Unpack it into /usr/src/linux-version-number, and establish a symbolic link named linux to it.
  3. cd linux; make xconfig.
  4. Select the various configuration parameters carefully.  In particular, focus on security related items.  E.g., "grsecurity".
  5. Build the kernel and the modules: make bzImage modules modules-install.
  6. Copy the bzImage, config and System.map files into /boot.
  7. Revise the /boot/grub/menu.lst so that the machine can be booted with the newly built kernel.
  8. Reboot into the new kernel.
  9. Experiment with various exploit programs, e.g., those of buffer overflow.
  10. Turnin a lab report detailing your config and observations.

Acknowledgements


References

  1. immunix.org  Carefully re-compiled RedHat distribution. Worth visiting.  Reference
  2. Seán Boran, Hardening Solaris, October 25, 1999, securityportal.com/cover/ coverstory19991025.html  Actually proper configuration and fortification. Reference.
  3. Michael Espinala, "The Hardening of Microsoft Windows NT Operating System Version 4.0", March 1998.  Proper configuration, actually.  Reference.
Tuesday October 28, 2003
Open Content Copyright © 2003 pmateti@cs.wright.edu