Chapter 2. Platform selection

Table of Contents

Gentoo Linux
Basic OS - the requirements
Access management services
Monitoring services
Backup services
Configuration management
Compliance management
Distributed resource management
Flows and feeds
Pluggable Authentication Modules
Principles behind PAM
How PAM works
Managing PAM configuration
Configuring PAM on the system
Learning more about PAM
Gentoo Hardened
Checking PaX and PIE/PIC/SSP results
SELinux as MAC
grSecurity kernel improvements
Using IMA and EVM
Key management
Securing OpenSSH
Using DNS SSHFP fields
Logging and auditing
System logging
Privilege escalation through sudo
Centralized sudoers file

Gentoo Linux

Within the reference architecture, Gentoo Linux is the standard. Standardization on a single platform allows organizations to keep the cost sufficiently low, but also offers the advantage that these solutions might be specific for the platform, rather than having to look for solutions that must support a multitude of platforms. Of course, the choice of picking Gentoo Linux here might seem weird - why not CentOS (as that has a possible commercial backing towards RedHat Enterprise Linux when needed)?

  • First of all - the author is a Gentoo Linux developer. Its the distribution he know the best.

    But in light of a (fictional) company, it might also be because its current (fictional) engineers are all Gentoo Linux developers, or because it has ties with regional Gentoo Linux supporting services. In light of many organizations, when there is choice between Linux distributions, one thing to consider is which distribution the engineers are most likely to work with. Alright, asking them will probably result in some heavy fighting to see which distribution is best (perhaps the Condorcet method can be used to find the best selection), but picking a distribution the engineers are less eager to support will result in bad administration anyhow.

  • The reason to use CentOS (RHEL) could be to have certified hosting of certain products which are only supported on RHEL (or similar). However, because the focus here is to use free software solutions, this is no requirement. But it is understandable that companies that do run proprietary software choose a distribution that is supported by their vendors.

  • Gentoo Linux offers a fairly flexible approach on supported features. Thanks to a good balance of USE flags, servers and services can be installed that offer just those services that are needed, without any additional dependencies or features that need to be disabled (in order to secure the services) anyhow. This leads to somewhat better performance, but also to a saving in storage requirements, patching frequency, etc. Gentoo is also quite fast in adopting new technologies, which might help the business stand out against the other competitors.

  • Gentoo uses rolling upgrades. That might not seem like a good way in enterprises, but it is. If an organization is doing things right, it is already distributing and rolling out patches and minor upgrades regularly. With Gentoo, this process is a bit more intrusive (as it might contain larger changes as well) but because the administrators are used to it, it is very much under control. As a result, whereas other organizations have to schedule large (expensive and time-consuming) upgrades every 3 to 5 years, Gentoo just moves along...

  • Gentoo has a subproject called Gentoo Hardened who strives to provide, implement and support security-improving patches on the base system. This project has always been a fore-runner in security-related risk mitigation strategies.

Of course, because this book is called "A Gentoo Linux Advanced Reference Architecture", it would be weird to have it talk about another distribution, wouldn't it?

Now, the selection of Gentoo Linux also has a few challenges up its sleeve.

  • Gentoo Linux is primarily a source-based distribution, which is frequently frowned upon in the enterprise market. Weirdly enough, enterprises don't find it strange that their development and operational teams keep on building frameworks and tools themselves because of lack of good tools. This is exactly where Gentoo Linux outshines the others: it offers many tools out-of-the-box to support every possible requirement.

    To reduce the impact of its source-only stigma, a chapter in this book is dedicated to the use of build servers and binhost support for improved manageability.

  • Because of its source-based nature, it also provides all the tools for malicious users to build exploits on the server itself.

    It is fairly easy to hide the compiler or at least have some group-based access control on it. But regardless of that - the moment a malicious user has (shell) access to a system, the system is screwed anyhow. It is fairly easy to transfer files (even full applications) towards the system then.

    To reduce possible impact here, a Mandatory Access Control system should be used which isolates processes and even users, confining them to just what they need to get their job done.

As architecture, focusing only on the x86_64 architecture (amd64) is beneficial, partially because it is the widest known in the Gentoo Linux development community, but also because its hardware is widely available and sufficiently cheap. It is also a processor architecture that is constantly evolving and has many vendors working on it (less monopolizing strategies) which makes it a better platform for consumers in my opinion.

That being said, this might be a good time to use a no-multilib approach in Gentoo Linux. Systems need to be fully x86_64 driven, partially for standardization as well, but also to make debugging easier. The fewer special cases that need to be thought about, the faster problems can be resolved. Generally though, this gives little (to no) additional advantage towards a multilib profile.

Basic OS - the requirements

When positioning an operating system platform such as Gentoo Linux, quite a few aspects already need to be considered in its design. It isn't sufficient to just create an image (or installation procedure) and be done with it. Basic services on operating systems need to be considered, such as backup/restore routines, updates & upgrades, etc. Most of the infrastructure needed to accomplish all that will be talked about further.


When managing multiple servers, some sort of centralized services are needed. This doesn't require a daemon/server architecture for all services though, as will be seen later on.

Figure 2.1. Services for an operating system platform

Services for an operating system platform

The mentioned services on the above drawing are quite basic services, which will need to be properly managed in order to get a well functioning environment.

Access management services

Access management services include:

  • Authentication (is the user indeed that user)

  • Authorization (is that user allowed to do the requested activity)

  • Auditing (what do we need to keep track of the users' actions)

In Gentoo Linux, the most probable component for authenticating users on the operating system level is OpenSSH. But in order to properly provide access services, not only the OpenSSH daemon itself is looked on, but also the centralized access management services (which will be OpenLDAP based).

Authorization on the operating system level is handled through the Linux DAC (Discretionary Access Control) and MAC (Mandatory Access Control) services. The DAC support is the most well-known. For MAC, we can use SELinux (Security Enhanced Linux).

Monitoring services

Monitoring services are used, not really to be informed when a service is down, but rather to quickly identify what is causing a service failure.

Service failures, like "I cannot resolve IP addresses" or "The web site is not reachable" are difficult to debug when lacking monitoring. Proper monitoring implementations allow to get an idea of the entire state of the architecture and its individual components. If monitoring sais that the web server processes are running and that remote web site retrieval agents are still pulling in site details, then there is most likely an issue with the connectivity between the client and the site (such as potential proxy servers or even networking or firewalls). On the other hand, if the monitoring shows that a web gateway daemon is not responsive, it is fairly easy to handle the problem as it is quite obvious where the problem lies.

Backup services

Although backups are not important, being able to restore stuff is - hence the need for backups ;-)

Even on regular servers, backups will be important to support fast recovery from human errors or application malpractices. Users, including administrators, make mistakes. Being able to quickly recover from deleted files or modifications will save hours of work later.

Configuration management

In order to properly update/upgrade the systems, as well as configure it to match the needs of the organization, some configuration management approach is needed. Whereas smaller deployments can be perfectly managed manually, decent configuration management allows to quickly deploy new systems, reconfigure systems when needed, support testing of configuration changes, etc.

Compliance management

In order for a system to be and remain secure, it is important to be able to validate configurations (compliance validation) as well as be able to scan systems for potential vulnerabilities and create an inventory of installed software. In this reference architecture, SCAP (Security Content Automation Protocol) services will be used supported through OpenSCAP.

Distributed resource management

In order to support automation tasks across multiple systems, JobScheduler services are used. This allows to combine tasks to automate more complex activities across systems.


In this reference architecture, the given services will be filled in with the following components.

Figure 2.2. Components for operating system platform

Components for operating system platform

The activities involved with those components are described in the next few sections.

Flows and feeds

A regular system has the following data flows enabled:

  1. Backup data is sent to the backup environment; the volumes of backups can be quite large, so take care to schedule backups during proper time windows.

  2. Logging data is sent towards a central log server, or a log collector.

Backup data

Backups can be quite large, and thus take a lot of network bandwidth. Depending on the backup method, it might also have performance impact on the server.

For these reasons, it is recommended that backups are scheduled outside the important activity windows.

Figure 2.3. Backup (cannot be more simpler than this ;-)

Backup (cannot be more simpler than this ;-)

Often, full backups taken every day are not optimal. Most backup solutions support full backup, differential backup (changes since last full or differential) and incremental backup (changes since last backup, regardless of type). The backup scheme then decides what to backup when, optimizing the backup volumes while keeping restore durations in mind.

  • Taking full backups every time requires large volumes, but restore of the data is rather quick (as no other backup set needs to be consulted).

  • Taking a single full backup and then incremental backups requires small volumes (except for the first full of course), but restore of the data will take some time as potentially all incremental backup sets need to be consulted

A possible backup scheme would be to

  • take a full backup on Saturday night

  • take a differential backup on Tuesday night

  • take incremental backups on the other dates

Also, keep in mind how long backups need to be retained. Backups might want to be kept for 1 month (around 4 full backups + remainder), but it might also be interesting to keep the first full backup of every month for an entire year, and the first full of each year (almost) eternally. It all depends on the retention requirements and pricing concerns (lots of backups requires lots of storage).

Logging data

Logging data is usually sent from the system logger (syslog) towards a central server. A central server is used as this allows to correlate events from multiple systems, as well as keep log management central.

Figure 2.4. Log flows from server to central log server

Log flows from server to central log server

The system logger receives its events from the /dev/log socket (well, there are a few other sources as well, but /dev/log is the most prominent one) from the various daemons. The local system logger is then configured to send the data to the central log server, and depending on the administrators' needs, locally as well.

If local logs are needed, make sure that the logs are properly rotated (using logrotate and a regular cron job).


To administer the system (and the components hosted on it), OpenSSH (for access to the system) and Puppet (for managing configuration settings) are used.

Figure 2.5. Operating system administration

Operating system administration

Standard operator/administrator access to the operating system is handled through the SSH secure shell. The OpenSSH daemon will be configured to use a central user repository for its authentication of users. This allows administrators to, for instance, change their password on a single system and ensure that the new password is then in use for all other systems as well. The SSH client is configured to download SSH fingerprints from the DNS server in case of a first-time connection to the server.

The configuration management will be handled through Puppet, whose configuration repository will be managed through a version-controlled system and pulled from the systems.


Systems (and the components and services that are hosted further) will be monitored through Icinga.

Figure 2.6. Operating system monitoring

Operating system monitoring

The Icinga agent supports various plugins that allow to monitor various aspects of the operating system and the services that run on it. The results of each "query" is then sent to the central Icinga database. The monitoring web interface, which is discussed later, interacts with the database to visually represent the state of the environment.


Considering this is a regular platform (with no additional services on yet), there is no specific operations defined yet.


For the user management on a Linux system, a central LDAP service for the end user accounts (and administrator accounts) is used. The functional accounts though (the Linux users under which daemons run) are defined locally. This ensures that there is no dependency on the network or LDAP for those services. However, for security reasons, it is important that these users cannot be used to interactively log on to the system.

The root account, which should only be directly used in case of real emergencies, should have a very complex password managed by a secured password management application.

End users are made part of one or more groups. These groups define the SELinux user assigned to them, and thus the rights they have on the system (even if they need root rights, their actions will be limited to those tasks needed for their role).


Additional services on the server are

  • compliance validation using openscap

  • inventory assessment using openscap

  • auditing through the Linux auditd daemon (and sent through the system logger for immediate transport)

  • host-based firewall using iptables (and managed through Puppet)

  • integrity validation of critical files

Compliance validation

To support a central compliance validation method, we use a local SCAP scanner (openscap) and centrally manage the configurations and results. This is implemented in a tool called pmcs (Poor Man Central SCAP).

Figure 2.7. Running compliance (and inventory) validation

Running compliance (and inventory) validation

The communication between the central server and the local server is HTTP(S) based.

Inventory management

As SCAP content is used to do inventory assessment, pmcs is used here as well.


Auditing on Linux systems is usually done through the Linux audit subsystem. The audit daemon can be configured to provide auditing functionalities on various OS calls, and most security conscious services are well able to integrate with auditd.

The important part still need to be covered is to send the audit events to a central server. The system logger is leveraged for this, and auditd configured to dispatch audit events to the local syslog.

Host-based firewall

A host-based firewall will assist in reducing the attack space towards the server, ensuring that network-reachable services are only accessed from (more or less) trusted locations.

Managing host-based firewall rules can be complex. We use the Puppet configuration management services to automatically provide the necessary firewall rules automatically.

Integrity validation of critical files

Critical files on the system are also checked for (possibly unwanted) manipulations. AIDE (Advanced Intrusion Detection Environment) can be used for this.

In order to do offline scanning (so that malicious software inside the host cannot meddle with the integrity validation scans) snapshotting is used on storage level and scanning is done on the hypervisor.

Pluggable Authentication Modules

Authentication management (part of access management) on a Linux server can be handled by PAM (Pluggable Authentication Modules). With PAM, services do not need to provide authentication services themselves. Instead, they rely on the PAM modules available on the system. Each service can use a different PAM configuration if it wants, although most of the time authentication is handled similarly across services. By calling PAM modules, services can support two-factor authentication out-of-the-box, immediately use centralized authentication repositories and more.

PAM provides a flexible, modular architecture for the following services:

  • Authentication management, to verify if a user is who it says it is

  • Account management, to check if that users' password has expired or if the user is allowed to access this particular service

  • Session management, to execute certain tasks on logon or logoff of a user (auditing, mounting of file systems, ...)

  • Password management, offering an interface for password resets and the like

Principles behind PAM

When working with PAM, administrators quickly find out what the principles are that PAM works with.

The first one is back-end independence. Applications that are PAM-aware do not need to incorporate any logic to deal with back-ends such as databases, LDAP service, password files, WS-Security enabled web services or other back-ends that have not been invented yet. By using PAM, applications segregate the back-end integration logic from their own. All they need to do is call PAM functions.

Another principle is configuration independence. Administrators do not need to learn how to configure dozens of different applications on how to interact with an LDAP server for authentication. Instead, they use the same configuration structure provided by PAM.

The final principle, which is part of the PAM name, is its pluggable architecture. When new back-ends need to be integrated, all the administrator has to do is to install the library for this back-end (by placing it in the right directory on the system) and configure this module (most of the modules use a single configuration file). From that point onward, the module is usable for applications. Administrators can configure the authentication to use this back-end and usually just need to restart the application.

How PAM works

Applications that want to use PAM link with the PAM library (libpam) and call the necessary functions that reflect the above services. Other than that, the application does not need to implement any specific features for these services, as it is all handled by PAM. So when a user wants to authenticate itself against, say, a web application, then this web application calls PAM (passing on the user id and perhaps password or challenge) and checks the PAM return to see if the user is authenticated and allowed access to the application. It is PAMs task underlyingly to see where to authenticate against (such as a central database or LDAP server).

Figure 2.8. Schematic representation of PAM

Schematic representation of PAM

The strength of PAM is that everyone can build PAM modules to integrate with any PAM-enabled service or application. If a company releases a new service for authentication, all it needs to do is provide a PAM module that interacts with its service, and then all software that uses PAM can work with this service immediately: no need to rebuild or enhance those software titles.

Managing PAM configuration

PAM configuration files are stored in /etc/pam.d and are named after the service for which the configuration applies. As the service name used by an application is often application-specific, you will need to consult the application documentation to know which service name it uses in PAM.


As the PAM configuration file defines how to authenticate users, it is extremely important that these files are very difficult to tamper with. It is recommended to audit changes on these files, perform integrity validation, keep backups and more.

Next to the configuration files, we also have the PAM modules themselves inside /lib/security or /lib64/security. These locations are often forgotten by administrators to keep track of, even though these locations are equally important as the configuration files. If an attacker can overwrite modules or substitute them with his own, then he also might have full control over the authentication results of the application.


As the PAM libraries are the heart of the authentication steps and methods, it too is extremely important to make it very difficult to tamper with. Again, auditing, integrity validation and backups are seriously recommended.

The PAM configuration files are provided on a per-application basis, although one application configuration file can refer to other configuration file(s) to use the same authentication steps. Let's look at a PAM configuration file for an unnamed service:

auth         required
auth         required

account      required

password     required

session      optional
session      required close
session      required
session      required level=audit
session      required open multiple
session      optional

Notice that the configuration file is structured in the four service domains that PAM supports: authentication, account management, password management and session management.

Each of the sections in the configuration file calls one or more PAM modules. For instance, sets the environment variable which can be used by subsequent modules. The return code provided by the PAM module, together with the control directive (required or optional in the above example), allow PAM to decide how to proceed.


The provided PAM module must succeed in order for the entire service (such as authentication) to succeed. If a PAM module fails, other PAM modules are still called upon (even though it is already certain that the service itself will be denied).


The provided PAM module must succeed in order for the entire service to succeed. Unlike required, if the PAM module fails, control is immediately handed back and the service itself is denied.


If the provided PAM module succeeds, then the entire service is granted. The remainder of the PAM modules is not checked. If however the PAM module fails, then the remainder of the PAM modules is handled and the failure of this particular PAM module is ignored.


The success or failure of this particular PAM module is only important if it is the only module in the stack.

Chaining of modules allows for multiple authentications to be done, multiple tasks to be performed upon creating a session and more.

Configuring PAM on the system

In order to connect the authentication of a system to a central LDAP server, the following lines need to be added in the /etc/pam.d/system-auth file (don't replace the file, just add the lines):

auth       sufficient  use_first_pass
account    sufficient
password   sufficient  use_authtok use_first_pass
session    optional

Also install the sys-auth/pam_ldap (and sys-auth/nss_ldap) packages.

A second step is to configure For /etc/ldap.conf, the following template can be used. Make sure to substitute the domain information with the one used in the environment:

suffix          "dc=genfic,dc=com"

bind_policy soft
bind_timelimit 2
ldap_version 3
nss_base_group ou=Group,dc=genfic,dc=com
nss_base_hosts ou=Hosts,dc=genfic,dc=com
nss_base_passwd ou=People,dc=genfic,dc=com
nss_base_shadow ou=People,dc=genfic,dc=com
pam_filter objectclass=posixAccount
pam_login_attribute uid
pam_member_attribute memberuid
pam_password exop
scope one
timelimit 2
uri ldap:// ldap:// ldap://

Secondly, /etc/openldap/ldap.conf needs to be available on all systems too:

BASE         dc=genfic, dc=com
URI          ldap:// ldap:// ldap://

Finally, edit /etc/nsswitch.conf so that other services can also use the LDAP server (next to central authentication):

passwd:         files ldap
group:          files ldap
shadow:         files ldap

Learning more about PAM

Most, if not all PAM modules have their own dedicated manual page.

$ man pam_env

Other information is easily available on the Internet, including:

Gentoo Hardened

To increase security of the deployments, all systems in this reference architecture will use a Gentoo Hardened deployment. Within the Gentoo Linux community, Gentoo Hardened is a project that oversees the research, implementation and maintenance of security-oriented projects in Gentoo Linux. It focuses on delivering viable security strategies for high stability production environments and is therefor absolutely suitable for this reference architecture.

Within this book's scope, all services are implemented on a Gentoo Hardened deployment with the following security measures in place:

  • PaX


  • SELinux as MAC

  • grSecurity kernel improvements

The installation of a Gentoo Hardened system is similar to a regular Gentoo Linux one. All necessary information can be found on the Gentoo Hardened project page.


The PaX project (part of grSecurity) aims to update the Linux kernel with defense mechanisms against exploitation of software bugs that allow an attacker access to the software's address space (memory). By exploiting this access, a malicious user could introduce or execute arbitrary code, execute existing code without the applications' intended behavior, or with different data than expected.

One of the defence mechanisms introduced is NOEXEC. With this enabled, memory pages of an application cannot be marked writeable and executable. So either a memory page contains application code, but cannot be modified (kernel enforced), or it contains data and cannot be executed (kernel enforced). The enforcement methods used are beyond the scope of this book, but are described online. Enforcing NOEXEC does have potential consequences: some applications do not work when PaX enforces this behavior. Because of this, PaX allows administrators to toggle the enforcement on a per-binary basis. For more information about this, see the Hardened Gentoo PaX Quickstart document (see resources at the end of this chapter). Note that this also requires PIE/PIC built code (see later).

Another mechanism used is ASLR, or Address Space Layout Randomization. This thwarts attacks that need advance knowledge of addresses (for instance through observation of previous runs). With ASLR enabled, the address space is randomized for each application, which makes it much more difficult to guess where a certain code (or data) portion is loaded, and as such attacks will be much more difficult to execute succesfully. This requires the code to be PIE built.

To enable PaX, the hardened-sources kernel in Gentoo Linux needs to be installed and configured according to the instructions found on the Hardened Gentoo PaX Quickstart document. Also install paxctl.

# emerge hardened-sources
# emerge paxctl


The given abbreviations describe how source code is built into binary, executable code.

PIC (Position Independent Code) is used for shared libraries to support the fact that they are loaded in memory dynamically (and without prior knowledge to the addresses). Whereas older methods use load-time relocation (where address pointers are all rewritten the moment the code is loaded in memory), PIC uses a higher abstraction of indirection towards data and function references. By building shared objects with PIC, relocations in the text segment in memory (which contains the application code) are not needed anymore. As such, these pages can be marked as non-writeable.

To find out if there are libraries that still support text relocations, install the pax-utils package and scan the libraries for text relocations:

# emerge pax-utils
$ scanelf -lpqt
TEXTREL  /opt/Citrix/ICAClient/

In the above example, the file is not built with PIC and as such could be more vulnerable to attacks as its code-containing memory pages might not be marked as non-writeable.

With PIE (Position Independent Executables) enabled, executables are built in a fashion similar to shared objects: their base address can be relocated and as such, PaX' ASLR method can be put in effect to randomize the address in use. An application binary that is PIE-built will show up as a shared object file rather than an executable file when checking its ELF header

$ readelf -h /bin/ls | grep Type
  Type:            DYN (Shared object file)

$ readelf -h /opt/Citrix/ICAClient/wfcmgr.bin | grep Type
  Type:            EXEC (Executable file)

SSP finally stands for Stack Smashing Protection. Its purpose is to add in additional buffers after memory allocations (for variables and such) which contain a cryptographic marker (often called the canary). When an overflow occurs, this marker is also overwritten (after all, that's how overflows work). When a function would return, this marker is first checked to see if it is still valid. If not, then an overflow has occurred and the application is stopped abruptly.

Checking PaX and PIE/PIC/SSP results

If the state of the system needs to be verified after applying the security measures identified earlier, install paxtest and run it. The application supports two modes: kiddie and blackhat. The blackhat test gives the worst-case scenario back whereas the kiddie-mode runs tests that are more like the ones script-kiddies would run. The paxtest application simulates certain attacks and presents plausible results to the reader.

A full explanation on the tests ran can be found in the /usr/share/doc/paxtest-*/README file.

# emerge paxtest
# paxtest blackhat

PaXtest - Copyright(c) 2003,2004 by Peter Busser <>
Released under the GNU Public Licence version 2 or later

Writing output to paxtest.log
It may take a while for the tests to complete
Test results:
PaXtest - Copyright(c) 2003,2004 by Peter Busser <>
Released under the GNU Public Licence version 2 or later

Mode: blackhat
Linux hpl 3.1.6-hardened #1 SMP PREEMPT Tue Dec 27 13:49:05 CET 2011 \
  x86_64 Intel(R) Core(TM) i5 CPU M 430 @ 2.27GHz GenuineIntel GNU/Linux

Executable anonymous mapping             : Killed
Executable bss                           : Killed
Executable data                          : Killed
Executable heap                          : Killed
Executable stack                         : Killed
Executable shared library bss            : Killed
Executable shared library data           : Killed
Writable text segments                   : Killed

These tests will try to write data and then execute it. The tests do this in different locations to verify if the memory protection measures are working. Killed means that it works as the attempt is stopped.

Executable anonymous mapping (mprotect)  : Killed
Executable bss (mprotect)                : Killed
Executable data (mprotect)               : Killed
Executable heap (mprotect)               : Killed
Executable stack (mprotect)              : Killed
Executable shared library bss (mprotect) : Killed
Executable shared library data (mprotect): Killed

These are virtually the same tests as before, but now the application first tries to reset or change the protection bits on the pages using mprotect.

Anonymous mapping randomisation test     : 33 bits (guessed)
Heap randomisation test (ET_EXEC)        : 13 bits (guessed)
Heap randomisation test (PIE)            : 40 bits (guessed)
Main executable randomisation (ET_EXEC)  : No randomisation
Main executable randomisation (PIE)      : 32 bits (guessed)
Shared library randomisation test        : 33 bits (guessed)
Stack randomisation test (SEGMEXEC)      : 40 bits (guessed)
Stack randomisation test (PAGEEXEC)      : 40 bits (guessed)

The randomisation tests try to find out which level of randomisation is put in place. Although randomisation by itself does not offer protection, it obscures the view malicious users have on the memory structures. The higher the randomisation, the better. On Gentoo Hardened, (almost) all binaries are PIE.

Return to function (strcpy)              : paxtest: return address \
                                           contains a NULL byte.
Return to function (memcpy)              : Vulnerable
Return to function (strcpy, PIE)         : paxtest: return address \
                                           contains a NULL byte.
Return to function (memcpy, PIE)         : Vulnerable

These types of attacks are very difficult to thwart by kernel protection measures only. The author of the paxtest application has put those in because he can, even though he knows PaX does not protect against them. In effect, he tries to show users that PaX is not an all-safe method and that additional security layers are still important.

SELinux as MAC

With a MAC (Mandatory Access Control), the system administrator can control which accesses are allowed and which not, and can enforce that the user cannot override this. Regular access patterns in Linux are discretionary, so the user can define this himself. In this book, SELinux is used as the MAC system. Another supported MAC in Gentoo Hardened is grSecurity's RBAC model.

Installing and configuring Hardened Gentoo with SELinux is described in the Gentoo SELinux handbook. It is seriously recommended to read through this resource a few times, as SELinux is not just about enabling a feature - it is a change in the security model and requires experience with it.

The SELinux strict policy (so no unconfined domains) is used for regular services, or MCS (without unconfined domains) when multi-tenancy support is needed.

$ id -Z

# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             strict
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              disabled
Policy deny_unknown status:     denied
Max kernel policy version:      26

grSecurity kernel improvements

Next to the previously mentioned grSecurity updates, grSecurity also adds in additional kernel protection measures. This includes additional hardening on chroot jails (to make it a lot more difficult to break out of a chroot) and file system abuse (like getting information from pseudo-filesystems to improve attacks).

For more information on enabling grSecurity, see the Gentoo grSecurity v2 Guide.

Using IMA and EVM

Another set of security subsystems available in recent Linux kernels are the Integrity Measurement Architecture and Extended Verification Modules subsystems.

With IMA, the integrity of files is validated (checksum or digital signature) against the recorded value in the extended attribute of that file. If the integrity matches, then the system allows to read (and if it isn't a digital signature, even modify) the file. If the checksum doesn't match, then the access to the file is prohibited.

EVM then ensures that the extended attributes of the file are not tampered with (as it stores the security information from IMA as well as SELinux information). This is accomplished using a HMAC value or a digital signature of the security related extended attributes. Because of the use of cryptographic methods, offline tampering of this data is not that simple - the attacker needs access to the key used by the HMAC or even the private key used for generating the signatures.

For more information on enabling IMA/EVM, see the Gentoo Integrity subproject documentation.


OpenSSH is the most popular secure shell daemon available, and is free software. For remotely managing Linux systems, we will use OpenSSH, but this also requires OpenSSH to be set up correctly (and securely).

Key management

Providing secure shells requires keys. In case of an SSH session, a handshake between the client and server occurs. During this handshake, key information is exchanged.

The server keys are stored in /etc/ssh and are usually generated by the system when OpenSSH is started for the first time.


This file provides the SSH version 1 (protocol version) RSA private key. SSH version 1 is considered insecure so should no longer be used.


This file provides the SSH version 2 DSA private key. DSA (Digital Signature Algorithm) is a standard for creating digital signatures; every SSH client will support this.


This file provides the SSH version 2 ECDSA private key. ECDSA (Elliptic Curve DSA) is a recent addition; not all SSH clients support it yet (although the SSH client provided by OpenSSH of course does)


This file provides the SSH versino 2 RSA private key. RSA is a market standard and thus also very well supported.

SSH also supports client certificates, a popular method for offering a more secure remote shell, as passwords can be leaked easily (or remembered by the wrong people) whereas SSH keys are a bit more difficult to memorize ;-) Recent OpenSSH versions also support chaining authentication methods.

Securing OpenSSH

There are a few guides available online to secure OpenSSH. Basically, these will recommend to

  • disable remote root logon

  • enable public key authentication

  • enable authentication chaining (first requires public key authentication, then password)

  • disable protocol version 1

  • enable PAM

Using DNS SSHFP fields

When an SSH client connects to an unknown server, it (depending on the configuration) will warn users about the server. It presents the SSH finger print (a checksum on the key) to the user, asking the user to validate that the key is indeed correct. This is needed to prevent a MITM (Man In the Middle) attack. Of course, most users do not check this (and some administrators would not even know how to check it themselves).

# ssh-keygen -l -f /etc/ssh/ssh_host_ecdsa_key
256    a1:0b:dd:5b:d0:38:ec:a2:26:fa:c1:74:06:fb:9c:c2  root@hpl (ECDSA)

To simplify this while remaining secure, we can add the SSH finger print data in the DNS records for the server (in an SSHFP field). The SSH client can then be configured to automatically trust the SSH fingerprint data in the DNS record (and to make it more secure - only when the records are signed properly through DNSSEC).

The SSHFP record can be generated using ssh-keygen:

# ssh-keygen -r hpl -f /etc/ssh/ssh_host_dsa_key
hpl IN SSHFP 2 1 68da815fa78336cbaf69eacad7b5c9ebf67f518

Logging and auditing

To handle auditing, the Linux audit daemon needs to be configured to send its audit events towards the central system logger, which is configured to forward these events as soon as possible to a central log server.

System logging

The syslog protocol is an IETF standard described in RFC5424. It offers a standard protocol for submitting log events across services. A log event is labeled with two important codes: the facility and the severity.

The facility defines what service the origin is of a log event. The supported facilities are:


security and authorization messages


security and authorization messages


system daemons


system scheduler messages


FTP daemon


printing messages


kernel messages


mail system


network news subsystem messages


messages generated internally by the system logger


user-level messages


UUCP subsystem


local use facilities

The severities supported are emergency, alert, critical, error, warning, notice, info and debug. Their description is provided by the previously mentioned RFC.

The syslog standard uses UDP as transport method. As such, message delivery across a network might not be guaranteed. Most system loggers support TCP-based message delivery as well nowadays.

How syslog works

Most, if not all system loggers use a socket called /dev/log through which local daemons can send syslog events. Scripts or applications that do not use the /dev/log socket can use the logger command to send events. The local system logger then writes the log events to the proper log files and/or sends it to remote system loggers.

Figure 2.9. Syslog mode of operations

Syslog mode of operations

System loggers can also be configured to receive syslog events from remote sources.

Balabit Syslog NG

Balabit's syslog-ng (provided through the app-admin/syslog-ng package) allows sending syslog events to a remote TCP-bound system logger through the following destination directive:

destination myloghost {
  tcp("" port(514));

Similarly, to have it receive syslog events, the following source directive can be used:

source src {
  tcp(port(514) keep-alive(yes));


To enable TCP syslog transmission on rsyslog (provided through the app-admin/rsyslog package):

$RuleSet remote
*.* @@

To enable it to receive syslog events:

$ModLoad imtcp
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514


Linux audit daemon interacts with the audit subsystem in the Linux kernel, and stores audit information in the necessary audit log files. The audit daemon can also dispatch audit events to other systems, including remote audit daemons.

Figure 2.10. Audit operations

Audit operations

On a Linux system, all system call activities can be audited and filters can be enabled to select what exactly (and when) auditing of activities is to be performed.

Configuring audit to syslog

The Linux audit daemon can send out the audit events to the system logger by using its audisp (Audit Dispatch) method. This is a plugin-like system in the Linux audit daemon that handles dispatching audit events based on certain rules.

To send all rules, edit /etc/audisp/plugins.d/syslog.conf and make sure active is set to "yes":

active = yes
direction = out
path = builtin_syslog
type = builting
args = LOG_INFO
format = string

Privilege escalation through sudo

With sudo (super user do) we can allow regular users to call certain commands with elevated privileges. To manage which commands are allowed by whom, we will also use the central LDAP server.

Centralized sudoers file



For more information about the topics in this chapter, more information is available at the following resources...

Gentoo Hardened: