Hacking Linux Exposed :: The Linux hacking and Windows hacking security Resource

Chrooting daemons and system processes

Written by LE Webmaster

Wednesday, 26 January 2005

The idea behind chroot is fairly simple. When you run BIND (or any other process) in a chroot jail, the process is simply unable to see any part of the filesystem outside the jail. For example, in this document, we'll set BIND up to run chrooted to the directory /chroot/named. Well, to BIND, the contents of this directory will appear to be /, the root directory. Nothing outside this directory will be accessible to it. You've probably encounted a chroot jail before, if you've ever ftped into a public system.

Table of Contents

Part I: Introduction to Chrooting

1.1 What is chrooting?
1.2 When is it appropriate to chroot?
1.3 Can all daemons be chrooted?
1.4 Will chrooting affect my users?
1.5 What is entailed in chrooting?

Part II: Requirements gathering

2.1 Can we chroot this daemon?
2.2 An Introduction to truss, lsof, and ldd
2.3 Determining datafile dependencies
2.4 Creating a datafile strategy
2.5 Determining library dependencies
2.6 Finding a good home for the jail

Part III: Setting up a chrooted environment

3.1 Creating an empty jail
3.2 Copying program and datafiles, configuring cron
3.3 Copying libraries
3.4 Creating devices
3.5 Altering startup scripts
3.6 The Final Product
3.7 Logging through syslog

Part IV: Advanced Topics

4.1 How to tell if you're chrooted
4.2 Breaking out of a chroot()ed cell
4.3 Frequent chroot() mistakes

Part I

1.1 What is chrooting?

The command/function chroot is short for 'change root', and is designed to
change the filesystem root for the environment it is applied to. This means the initial
slash (/) in any path names are made relative to the chrooted path. For
example, if a file called:
/home/jonz/hello.txt
exists on the system, and then I chrooted to /home/jonz, the file would then exist, in my chrooted environment, as:
/hello.txt

The purpose of chrooting is designed to create an impenetrable (theoretically) "jail"
protecting what is being chrooted from being able to read or modify any files
outside of the chrooted environment. In the example above, I would be unable
to access any files outside of /home/jonz, since / is now pointing to
/home/jonz. Chrooting is commonly used to jail users in multiuser environments to protect system files. Chrooting
can also be used to jail system daemons to help prevent them
from being viable targets for hackers. If a hacker should exploit a
vulnerability in a chrooted system daemon, their ability to affect files
outside of the jail, or obtain a root shell is significantly more difficult.
One big reason for this is that a shell is no longer part of the environment's
path, so even if the hacker blows the stack away there's no shell to drop to.
Many people have claimed to be able to break out of a chrooted jail, but in
many cases it was from a shell (which doesn't exist in our case). Breaking
out of a daemon-environment jail is at the very least, extremely difficult.

1.2 When is it appropriate to chroot daemons or system processes?

Chrooting daemons is a practical method of adding an additional layer of
security to your system. Many system processes and third party applications
already have some safeguards against vulnerability exploits. Many tools now
have the ability to run as non-root users, which makes it harder for hackers
to attack root.

Network security layers such as firewalls, TCP wrappers, filters, and etcetera
also add to the overall security of a system. Like all of these, chrooting is
appropriate for most implementations, where it is possible to do so without
compromising functionality.

1.3 Can all daemons be chrooted?

Technically you can chroot anything you like including your mother's casserole, but in some cases chrooting is not always possible without "breaking" something,
or in other circumstances, without an elaborate nonconventional configuration
not worth the trouble . Some daemons cannot function properly in a chrooted environment due to the
complexity of their functions. For example, sendmail must have the ability to
access users' home directories to search for .forward files. There is no
practical way to chroot sendmail without creating an elaborate, time consuming
"mirror". This is why sendmail has an alternative solution, smrsh (sendmail
restricted shell). A majority of system daemons, however, can be safely
chrooted with little effort.

1.4 Will chrooting affect my users?

If it is done correctly, your users should notice no difference in system
behavior. Chrooting in itself will not directly affect your users or
alter your operating system. The existing system is commonly left untouched,
while small "jails" are created to provide services from. Keep in mind also,
we're not talking about taking advantage of chroot packages existing in
ftp or ssh daemons. That is a similar, yet different concept than the one
we're discussing here. We're discussing chrooting the system daemons that
provide these services to transparently be able to get their job done with
additional security.

1.5 What is entailed in chrooting?

Chrooting any daemon is a process and commonly involves the following
steps to producing a viable chrooted daemon:

Creating a "jail" directory
Copying the daemon and any required datafiles
Copying required system libraries
Altering any startup scripts to launch the daemon in the new environment

Steps 2 and 3 commonly consume most time. Step 2, in particular, requires a
certain level of creativity in some circumstances. For example, a mail
popper has to have access to the mail folders on the system. Copying the
mail folders to the jail will not produce the desired results, so we then
must create a jail "around" the mail folders. We'll get into the complexities
of this later.

Part II

2.1 Can we chroot this daemon?

Before even creating a directory or copying a single file, we need to
ask ourselves some important questions. The first is whether or not the
daemon we're looking at can be chrooted. We're going to be working
with three examples of system daemons below, and pose the same key questions:

Does the daemon access files in one general location?

Is it viable to copy the files either once, or periodically using cron?

Is it prudent to create a jail "around" a specific set of files?

If you answered 'No' to all three of these questions, it may not be a good idea to chroot this daemon. Lets look at the following examples:

Eudora Qpopper

Does the daemon access files in one general location?

Primarily, a mail popper needs to access the users' mail folders. There are
some other files such as some /etc files required to perform authentication.
The answer to this question is 'YES' as a majority of the files needed are
in one location (/var/mail).

Is it viable to copy the files either once, or periodically using cron?

Copying mail folders is never a good idea, and copying them only once will end
up giving our users stale mailboxes. Thus, it's not a good idea to copy these
files at all. It is prudent, however, to copy the /etc files used and
any other files used once per day or more/less frequently as needed. Since
we are trying to protect these files, we don't want to give a potential
hacker access to the originals.

Is it prudent to create a jail "around" a specific set of files?

There are two alternatives when dealing with files in a jail. Either move
or copy the files into the jail, or build your jail around the files. In the
case of mail, it is extremely resource intensive to copy mail folders back
and forth every few minutes, and it is just as inefficient to try and maintain
a directory full of symbolic links. It is
far more efficient to get the popper to use the actual mail folders
in their original location. Our answer here is 'YES'.

Sun's RPCBIND

Does the daemon access files in one general location?

The daemon primarily accesses several small /etc files.

Is it viable to copy the files either once, or periodically
using cron?

Most of the files accessed are small and scarcely updated. It would be very
viable to copy these files to a different directory once a day.

Is it prudent to create a jail "around" a specific set of files?

Unlike our mail popper, which accessed primarily mail, /etc contains many of
the critical system files we're trying to protect. It would be very BAD and
defeat the purpose of chrooting in the first place to give a potential
hacker access to any files in /etc. Thus, it would be a good idea to chroot
THIS daemon by having a totally separate jail, and simply copy the files over.

Sendmail

Does the daemon access files in one general location?

No. Sendmail must have access to several files in several different locations.
This includes users' home directories and other files. Simultaneously,
sendmail must also be able to access the mail spool (/var/spool/mqueue) and
user mail directories (/var/mail).

Is it viable to copy the files either once, or periodically
using cron?

In order to create a chrooted environment containing all the files necessary
for sendmail to function, user directories would need to be mirrored in a
secure location to contain the files necessary to redirect and manupulate mail.
This would also require the creation of new directories and the deletion of
old ones when user accounts and manipulated. Simultaneously, sendmail must
also be able to access the mail spool and user mail folders. It is not
viable to copy any of these in and out.

Is it prudent to create a jail "around" a specific set of files?

Lets take a look at this. We could create a jail around the mail folders,
but then we'd lose the ability to write to the mail spool. We could jail
/var, allowing access to both but then we would be allowing the chrooted
environment access to potentially critical files. Neither solutions address
the other issue of having to create a mirror for user directories. There
doesn't seem to be any one set of files we could create a jail around.

Sendmail, as the answers suggest, is not a good candidate
for chrooting. Although it may be possible, the setup would be
elaborate and resource-intensive. It's a good idea to just stick to the
security that comes with sendmail.

2.2 Introduction to truss, lsof, and ldd

truss (or strace)

If you 'man truss' you'll see that truss is a tool designed to trace system
calls and signals. strace is a similar tool used in Linux.
System calls include calls to libraries and accessing
files. Truss is a great way to find out what resources a particular daemon
is using. To use truss, you simply type 'truss' followed by the command
you would normally use to start the daemon. Output you are looking for will
look like this:

open("/usr/lib/libc.so.1", O_RDONLY) = 3

Program opened the libc library

open64("/usr/local/etc/my.config", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3

Program opened /usr/local/etc/my.config for writing

Running truss on a daemon should immediately give you a long list of files to
start with on your search. It's a good idea to utilize all the different
features and commands of a daemon and watch for new files in truss. Without
truss, you'll end up breaking your chrooted jail once for every time you're
missing a data file.

lsof

lsof is another great tool for finding the resources a daemon uses. lsof
is designed to operate on daemons that are already running. It will show you
open files, network connections, and much more. lsof does not come with
most systems, as it is a third party tool. It can be downloaded from many
sites such as www.sunfreeware.com. Sources are readily available via
anonymous ftp from ftp://vic.cc.purdue.edu/pub/tools/unix/lsof.

ldd

ldd is a tool handy for determing what library resources your program will
need in order to execute. ldd stands for 'list dynamic dependencies'. Running
ldd followed by the path and filename of the daemon you are chrooting will give
you a list of libraries you must plan to copy over to the jail.

A note on dynamic libraries: If you have access to the
sources for the program you are chrooting, you may wish to compile it
with static libraries. This will prevent you from having to copy over
libraries. If you use an operating system whose manufacturer updates
the libraries frequently, however, you may wish to remain dynamic.

2.3 Determining datafile dependencies

The first little exercise we went through in 2.1 got us thinking about what
datafiles the daemon is dependent on. Some are relocatable, while others
we have to build a jail around. Only two of the three examples from the
previous section were viable (Eudora Qpopper and Sun's RPCBIND). Both are
very good candidates for chrooting and both have had severe security
vulnerabilities in the past. Now lets move on to mapping out the actual
datafiles that we're going to need to move or build around for these two
examples. It's perfectly alright to miss a few small files as we can always
find a solution for them before we turn the new service up. There are several
ways to determine datafiles used. These include, but are not limited to:

The obvious (duh, a mail popper accesses mail folders)
Examining configuration files
Using tools such as 'lsof' and 'truss'
(Preferably not) Breaking the daemon later on in its jail

It's important to account for as many important datafiles as you can. Looking at
our two examples, we're able to create a pretty good list of the files we're going
to need:

Eudora Qpopper:

/var/mail/* (Inside jail)

The following files from /etc (copied in periodically)

<br/>hosts.allow<br/>hosts.deny<br/>netconfig<br/>nsswitch.conf<br/>passwd<br/>resolv.conf<br/>services<br/>shadow<br/>

Sun RPCBIND

The following files from /etc (copied in periodically)

<br/>hosts<br/>netconfig<br/>nsswitch.conf<br/>resolv.conf<br/>services<br/>

Timezone from /usr/local/zoneinfo

A tmp directory

2.4 Creating a datafile strategy

Once you have a good list of files that will be used, keep track of how you
plan on getting them into your jail. Will you copy them in? How often? Once or
daily? Is your jail going to be built around them? Building a strategy will
tell you which files are going to need to be copied, which ones you
will build the jail around, and will start to create an idea in your head
of where you want this jail to go. It is imperative that you do not
build your jail around any system critical files, such as /etc, as this will
alleviate the purpose of a jail. If you must build a jail around something,
build it around a directory that would not necessarily cause a system failure
if hacked.

If you draw a circile on paper, you can then draw all the files outside of
the jail outside of the circle. Draw lines into the circle and annotate how
they will be moved or copied, including how often. Any files you plan
on building your jail around should be drawn inside the circle. This will
start to show you how big your jail is going to need to be (filesystem
path-wise) and give you some good notes to work with when it's time to
create the jail.

2.5 Determining library dependencies

All the program code that your daemon uses to execute is not necessarily
included in just the program file. Many peices of shared code are re-used
in the form of libraries. These libraries are required by the program in
order to execute properly. In a chrooted environment, the program will
not have direct access to these libraries in their original locations. They
will need to be initially copied into the jail in order for the program to
function. The best way to determine what libraries your daemon will use
is by using the command 'ldd'. ldd will list the dynamic dependencies
(commonly shared libraries) required in order to execute. Take note of these,
and make plans to copy them into your new jail.

Document the libraries used just as you did with your datafiles. You already
know how they're going to be copied into the jail (using cp, and only once).
It may be a good idea to keep this information handy in the event that you
upgrade or patch your operating system, you may want to copy the updated
libraries to their jail locations. An alternative would be to create a
cron job to copy them over once a month.

In our examples, we've extracted the following libraries using truss:

Eudora Qpopper

<br/>/usr/lib/libnsl.so.1<br/>/usr/lib/libsocket.so.1<br/>/usr/lib/libresolv.so.2<br/>/usr/lib/libmail.so.1<br/>/usr/lib/librt.so.1<br/>/usr/lib/libcrypt_i.so.1<br/>/usr/lib/libc.so.1<br/>/usr/lib/libdl.so.1<br/>/usr/lib/libmp.so.2<br/>/usr/lib/libaio.so.1<br/>/usr/lib/libgen.so.1<br/>/usr/lib/libpthread.so.1<br/>/usr/lib/libthread.so.1<br/>

Sun RPCBIND

<br/>/usr/lib/libsocket.so.1<br/>/usr/lib/libnsl.so.1<br/>/usr/lib/libdl.so.1<br/>/usr/lib/libc.so.1<br/>/usr/lib/libmp.so.2<br/>

2.6 Finding a good home for your jail

You now have all the information necessary to determine where to set up
your jail. If your jail is going to be 'built around' existing files (such as
our Eudora Qpopper example), you will need to create it offset to that directory.
If you do not need to create it around any files, choose an arbitrary location
with significant disk space. We are going to use
/var/mail for our popper and /var/rpcbind
for Sun RPCBIND. Remember, your programs are not going to be know they're jailed
so they're going to try and access the files just as they would normally which means
they better either be where the daemon expects them to be (relatively) or symlinked.
You're now ready to move onto Part III!

Part III

3.1 Creating an empty jail

The first step to creating your empty jail is to create a set of directories
to hold your daemon, data files, and libraries. Lets first attack our
Sun RPCBIND example. We've chosen /var/rpcbind
as our jail home. Here's how we'll set up our directory structure:

<br/>mkdir /var/rpcbind<br/>mkdir /var/rpcbind/dev<br/>mkdir /var/rpcbind/etc<br/>mkdir /var/rpcbind/tmp<br/>mkdir -p /var/rpcbind/usr/lib<br/>

Since this daemon will be chrooted, it wil need access to certain /dev devices,
some datafiles in etc, its own tmp directory, and a library directory. These
will all be filled with appropriate files by the time we're done.

Now onto our Eudora Qpopper example. Since qpopper's jail is going to be built
around our mailspool, the jail will need to start in /var/mail. We need to
go ahead and create our initial directories:

<br/>mkdir /var/mail/dev<br/>mkdir /var/mail/etc<br/>mkdir -p /var/mail/usr/lib <br/>mkdir -p /var/mail/usr/local/lib<br/>mkdir -p /var/mail/usr/sbin<br/>mkdir -p /var/mail/usr/share/lib/zoneinfo/US<br/>

But we're not done yet. Our popper is expecting the mail to be in /var/mail. When
it's chrooted, the path / is /var/mail...so when it tries to access /var/mail it
will get a file not found! How do we circumvent this problem? We need to symlink
/var/mail/var/mail to /var/mail which is, relatively speaking, / ...

<br/>mkdir /var/mail/var<br/>ln -s / /var/mail/var/mail <br/>

Now when our chrooted popper goes to access /var/mail, it'll be pointed to /, which is the real
/var/mail!

Our final products should look like:

Sun RPCBIND

<br/><br/>drwxr-xr-x   7 root     other        512 Aug  1 18:31 ./<br/>drwxr-xr-x  34 root     sys          512 Aug  1 18:07 ../<br/>drwxr-xr-x   2 root     other        512 Aug  1 18:16 dev/<br/>drwxr-xr-x   2 root     other        512 Aug  1 18:10 etc/<br/>drwxr-xr-x   2 root     other        512 Aug  1 18:31 tmp/<br/>drwxr-xr-x   6 root     other        512 Jul 28 15:55 usr/<br/>drwxr-xr-x   3 root     other        512 Aug  1 18:19 var/<br/>

Eudora Qpopper

<br/><br/>drwxrwxrwt   7 root     mail         512 Oct 18 19:36 ./<br/>drwxr-xr-x  34 root     sys          512 Aug  1 18:07 ../<br/>drwxr-xr-x   2 root     other        512 Jul 29 13:33 dev/<br/>drwxr-xr-x   2 root     other        512 Jul 28 16:18 etc/<br/><br/>            - Some mailboxes mixed in here -<br/><br/>drwxr-xr-x   6 root     other        512 Jul 28 15:55 usr/<br/>drwxr-xr-x   2 root     other        512 Aug  1 18:06 var/<br/>

3.2 Copying program and data files, configuring cron

Now that our directory structure is set up, lets copy over our program and data files.
Since we're going to be running rpcbind from an init script, it's not necessary to
copy the actual binary over for that. We will, hoever, need to copy the popper
over as it will be run from inetd. If inetd expects the file to be in /usr/sbin,
you will need to create a usr/sbin in your jail (/var/mail/usr/sbin) and copy
the daemon's executable to there. Copy all the /etc files we previously discussed
over and make sure the permissions are identical. For files you will want to update
frequently, create cron jobs to update them. When you're finished, your structure
should look similar to this:

Sun RPCBIND

<br/><br/>% ls -l /var/rpcbind/etc<br/>-r--r--r--   1 root     other         90 Jul 28 16:18 hosts<br/>-rw-r--r--   1 root     other       1239 Jul 28 16:09 netconfig<br/>-rw-r--r--   1 root     other        835 Jul 28 15:32 nsswitch.conf<br/>-rw-r--r--   1 root     other        140 Jul 28 16:14 resolv.conf<br/>-r--r--r--   1 root     other       3649 Jul 28 16:14 services<br/>

Eudora Qpopper

<br/><br/>% ls -l /var/mail/etc<br/><br/>-r--r--r--   1 root     other         90 Jul 28 16:18 hosts<br/>-rw-------   1 root     other         73 Oct 18 19:15 hosts.allow<br/>-rw-------   1 root     other          9 Jul 28 15:58 hosts.deny<br/>-rw-r--r--   1 root     other       1239 Jul 28 16:09 netconfig<br/>-rw-r--r--   1 root     other        835 Jul 28 15:32 nsswitch.conf<br/>-r--r--r--   1 root     other        815 Oct 18 19:15 passwd<br/>-rw-r--r--   1 root     other        140 Jul 28 16:14 resolv.conf<br/>-r--r--r--   1 root     other       3649 Jul 28 16:14 services<br/>-r--------   1 root     other        502 Oct 18 19:15 shadow<br/>

The cron jobs we came up with look like this. Whatever works for you...

<br/>0,30 * * * * cp -p /etc/passwd /var/mail/etc/passwd<br/>0,30 * * * * cp -p /etc/shadow /var/mail/etc/shadow<br/>0 0 * * * cp -p /etc/hosts.* /var/mail/etc/<br/>0 0 0 * * cp -p /etc/services /var/rpcbind/etc<br/>0 0 0 * * cp -p /etc/resolv.conf /var/mail/etc<br/>

3.3 Copying libraries

Remember those libraries we discussed? We need to copy them to their relative
places in the jail as well. Most libraries can be found in /usr/lib although
some exist in /usr/local/lib.
In the same fashion as your datafiles, copy the libraries over with identical
permissions. You should end up with something similar to this (depending on
your operating system):

<br/>% ls -l /var/rpcbind/usr/lib<br/><br/>-rwxr-xr-x   1 root     other     200292 Jul 28 15:27 ld.so.1*<br/>-rwxr-xr-x   1 root     other      41628 Jul 28 15:28 libaio.so.1*<br/>-rwxr-xr-x   1 root     other     938940 Jul 28 15:28 libc.so.1*<br/>-rwxr-xr-x   1 root     other      15616 Jul 28 15:27 libcrypt_i.so.1*<br/>-rwxr-xr-x   1 root     other       4448 Jul 28 15:28 libdl.so.1*<br/>-rwxr-xr-x   1 root     other      40540 Jul 28 15:28 libgen.so.1*<br/>-rwxr-xr-x   1 root     other      29548 Jul 28 15:27 libmail.so.1*<br/>-rwxr-xr-x   1 root     other      19584 Jul 28 15:28 libmp.so.2*<br/>-rwxr-xr-x   1 root     other     730672 Jul 28 15:27 libnsl.so.1*<br/>-rwxr-xr-x   1 root     other      35308 Jul 28 15:57 libpthread.so.1*<br/>-rwxr-xr-x   1 root     other     326336 Jul 28 15:27 libresolv.so.2*<br/>-rwxr-xr-x   1 root     other      39048 Jul 28 15:27 librt.so.1*<br/>-rwxr-xr-x   1 root     other      65876 Jul 28 15:27 libsocket.so.1*<br/>-rwxr-xr-x   1 root     other     166624 Jul 28 15:58 libthread.so.1*<br/>-rwxr-xr-x   1 root     other      19648 Jul 28 16:17 nss_dns.so.1*<br/>-rwxr-xr-x   1 root     other      38832 Jul 28 15:33 nss_files.so.1*<br/>-rwxr-xr-x   1 root     other      38292 Jul 28 15:33 nss_nis.so.1*<br/>-rwxr-xr-x   1 root     other      12284 Aug  1 18:15 straddr.so*<br/><br/>% ls -l /var/mail/usr/lib /var/mail/usr/local/lib<br/><br/>-rwxr-xr-x   1 root     other     200292 Jul 28 15:27 ld.so.1*<br/>-rwxr-xr-x   1 root     other      41628 Jul 28 15:28 libaio.so.1*<br/>-rwxr-xr-x   1 root     other     938940 Jul 28 15:28 libc.so.1*<br/>-rwxr-xr-x   1 root     other      15616 Jul 28 15:27 libcrypt_i.so.1*<br/>-rwxr-xr-x   1 root     other       4448 Jul 28 15:28 libdl.so.1*<br/>-rwxr-xr-x   1 root     other      40540 Jul 28 15:28 libgen.so.1*<br/>-rwxr-xr-x   1 root     other      29548 Jul 28 15:27 libmail.so.1*<br/>-rwxr-xr-x   1 root     other      19584 Jul 28 15:28 libmp.so.2*<br/>-rwxr-xr-x   1 root     other     730672 Jul 28 15:27 libnsl.so.1*<br/>-rwxr-xr-x   1 root     other      35308 Jul 28 15:57 libpthread.so.1*<br/>-rwxr-xr-x   1 root     other     326336 Jul 28 15:27 libresolv.so.2*<br/>-rwxr-xr-x   1 root     other      39048 Jul 28 15:27 librt.so.1*<br/>-rwxr-xr-x   1 root     other      65876 Jul 28 15:27 libsocket.so.1*<br/>-rwxr-xr-x   1 root     other     166624 Jul 28 15:58 libthread.so.1*<br/>-rwxr-xr-x   1 root     other      19648 Jul 28 16:17 nss_dns.so.1*<br/>-rwxr-xr-x   1 root     other      38832 Jul 28 15:33 nss_files.so.1*<br/>-rwxr-xr-x   1 root     other      38292 Jul 28 15:33 nss_nis.so.1*<br/>

3.4 Creating Devices

Here comes the tricky part. With the daemon chrooted, it will not be able
to access the traditional /dev directory, which includes several commonly
used devices such as /dev/null, /dev/log, and etcetera. Finding out what
devices your daemon uses is no walk in the park either. You can find most
out by using truss and sometimes strings | grep dev, but for the most part
you'll have to watch for error messages if you are missing a device. There
are a general set of devices that should be created for any daemon:

<br/><br/>/dev/conslog<br/>/dev/log<br/>/dev/msglog<br/>/dev/null<br/>/dev/tcp<br/>/dev/ticlts<br/>/dev/ticots<br/>/dev/ticotsord<br/>/dev/udp<br/>/dev/zero<br/>

Here's how to create your devices (do this for each device):

1. do an 'ls -lL /dev/[devicename]'. The output will look like this:

crw-rw-rw- 1 root sys 13, 2 Oct 18 19:56 /dev/null

The number in red is your 'major' number. The number in blue is your 'minor' number. The first letter, which is in green
will be either a 'c' or a 'b' signifying whether the device is a character or block device. With all this information,
you can now create the node.

2. cd into the jail's dev directory (e.g. /var/mail/dev or /var/rpcbind/dev). Use the command 'mknod' to create the device. For example: mknod null c 13 2 would create the device we listed in our example.
Most machines have different major/minor numbers so be careful.

3. using chown and chmod, make sure the device reflects identical permissions as the original device.

For our Sun RPCBIND example, we ended up breaking it later on to discover that the creation of a directory called
rpc_door with the sticky bit set needed to be created. This may not be the case on your operating system, but if
you come across something like that you'll need to create it:

<br/>mkdir -p /var/rpcbind/var/run/rpc_door<br/>chmod +t /var/rpcbind/var/run/rpc_door<br/>

Blah.

3.5 Altering startup scripts

Now you should be done! If you missed something, you'll likely hear about it when
you first run the daemon to test it. You did test it didn't you? Oh wait, we forgot
to tell you how! Well, before you can safely test most daemons, you need to turn their
non-chrooted version off. Once you've killed the production daemon, you can execute
the command like this:

/usr/sbin/chroot /var/rpcbind /usr/sbin/rpcbind

Hopefully everything should run fine. If you get complaints of missing files or libraries,
you can use truss to determine the filename, and copy it into your path. Once you're
up and running, you can simply comment out the old startup command in your startup scripts
and replace it with a chrooted version.

Chrooting from inetd

Eudora Qpopper runs from inetd, and therefore needs its own commandline to start it up
chrooted every time. To do this, use a line similar to:

pop3 stream tcp nowait root /usr/sbin/chroot chroot /var/mail /usr/sbin/in.pop3

Or if you're using TCP wrappers, you'll need to "trick" inetd into thinking
its service name is still in.pop3 (or whatever you use). In order to do this,
symlink your service name to chroot and then use the service name in inetd.conf.
For example:

lrwxrwxrwx 1 root other 6 Jul 28 15:42 /usr/sbin/in.pop3 -> chroot*

pop3 stream tcp nowait root /usr/sbin/tcpd in.pop3 /var/mail /usr/sbin/in.pop3

/usr/sbin/in.pop3 is *actually* /usr/sbin/chroot (symlinked), so when inetd
starts the service in.pop3, chroot actually runs with the parameters '/var/mail /usr/sbin/in.pop3' which will run the chrooted version of in.pop3 located in
/var/mail.
When the chrooted copy is started up by chroot, the service name will be set
correctly.

3.6 The Final product

Our final, working projects looks like this:

Sun RPCBIND

<br/><br/>d none /var/rpcbind 0755 root other<br/>d none /var/rpcbind/dev 0755 root other<br/>c none /var/rpcbind/dev/conslog 21 0 0666 root other<br/>c none /var/rpcbind/dev/log 21 5 0640 root other<br/>c none /var/rpcbind/dev/msglog 97 1 0600 root other<br/>c none /var/rpcbind/dev/null 13 2 0666 root other<br/>c none /var/rpcbind/dev/udp 41 0 0666 root other<br/>c none /var/rpcbind/dev/tcp 42 0 0666 root other<br/>c none /var/rpcbind/dev/ticlts 105 2 0666 root other<br/>c none /var/rpcbind/dev/ticotsord 105 1 0666 root other<br/>c none /var/rpcbind/dev/ticots 105 0 0666 root other<br/>d none /var/rpcbind/var 0755 root other<br/>d none /var/rpcbind/var/run 0755 root other<br/>d none /var/rpcbind/var/run/rpc_door 1777 root root<br/>d none /var/rpcbind/tmp 0755 root other<br/>d none /var/rpcbind/usr 0755 root other<br/>d none /var/rpcbind/usr/share 0755 root other<br/>d none /var/rpcbind/usr/share/lib 0755 root other<br/>d none /var/rpcbind/usr/share/lib/zoneinfo 0755 root other<br/>d none /var/rpcbind/usr/share/lib/zoneinfo/US 0755 root other<br/>f none /var/rpcbind/usr/share/lib/zoneinfo/US/Eastern 0644 root bin<br/>d none /var/rpcbind/usr/lib 0755 root other<br/>f none /var/rpcbind/usr/lib/ld.so.1 0755 root other<br/>f none /var/rpcbind/usr/lib/libnsl.so.1 0755 root other<br/>f none /var/rpcbind/usr/lib/libsocket.so.1 0755 root other<br/>f none /var/rpcbind/usr/lib/libresolv.so.2 0755 root other<br/>f none /var/rpcbind/usr/lib/libmail.so.1 0755 root other<br/>f none /var/rpcbind/usr/lib/librt.so.1 0755 root other<br/>f none /var/rpcbind/usr/lib/libcrypt_i.so.1 0755 root other<br/>f none /var/rpcbind/usr/lib/libc.so.1 0755 root other<br/>f none /var/rpcbind/usr/lib/libdl.so.1 0755 root other<br/>f none /var/rpcbind/usr/lib/libmp.so.2 0755 root other<br/>f none /var/rpcbind/usr/lib/libaio.so.1 0755 root other<br/>f none /var/rpcbind/usr/lib/libgen.so.1 0755 root other<br/>f none /var/rpcbind/usr/lib/nss_files.so.1 0755 root other<br/>f none /var/rpcbind/usr/lib/nss_nis.so.1 0755 root other<br/>f none /var/rpcbind/usr/lib/libpthread.so.1 0755 root other<br/>f none /var/rpcbind/usr/lib/libthread.so.1 0755 root other<br/>f none /var/rpcbind/usr/lib/nss_dns.so.1 0755 root other<br/>f none /var/rpcbind/usr/lib/straddr.so 0755 root other<br/>d none /var/rpcbind/usr/sbin 0755 root other<br/>f none /var/rpcbind/usr/sbin/rpcbind 0555 root other<br/>d none /var/rpcbind/usr/local 0755 root other<br/>d none /var/rpcbind/usr/local/sbin 0755 root other<br/>d none /var/rpcbind/usr/local/lib 0755 root other<br/>d none /var/rpcbind/etc 0755 root other<br/>f none /var/rpcbind/etc/nsswitch.conf 0644 root other<br/>f none /var/rpcbind/etc/netconfig 0644 root other<br/>f none /var/rpcbind/etc/services 0444 root other<br/>f none /var/rpcbind/etc/resolv.conf 0644 root other<br/>f none /var/rpcbind/etc/hosts 0444 root other<br/>

Eudora Qpopper

<br/><br/>d none /var/mail 1777 root mail<br/>d none /var/mail/usr 0755 root other<br/>d none /var/mail/usr/share 0755 root other<br/>d none /var/mail/usr/share/lib 0755 root other<br/>d none /var/mail/usr/share/lib/zoneinfo 0755 root other<br/>d none /var/mail/usr/share/lib/zoneinfo/US 0755 root other<br/>f none /var/mail/usr/share/lib/zoneinfo/US/Eastern 0644 root bin<br/>d none /var/mail/usr/lib 0755 root other<br/>f none /var/mail/usr/lib/ld.so.1 0755 root other<br/>f none /var/mail/usr/lib/libnsl.so.1 0755 root other<br/>f none /var/mail/usr/lib/libsocket.so.1 0755 root other<br/>f none /var/mail/usr/lib/libresolv.so.2 0755 root other<br/>f none /var/mail/usr/lib/libmail.so.1 0755 root other<br/>f none /var/mail/usr/lib/librt.so.1 0755 root other<br/>f none /var/mail/usr/lib/libcrypt_i.so.1 0755 root other<br/>f none /var/mail/usr/lib/libc.so.1 0755 root other<br/>f none /var/mail/usr/lib/libdl.so.1 0755 root other<br/>f none /var/mail/usr/lib/libmp.so.2 0755 root other<br/>f none /var/mail/usr/lib/libaio.so.1 0755 root other<br/>f none /var/mail/usr/lib/libgen.so.1 0755 root other<br/>f none /var/mail/usr/lib/nss_files.so.1 0755 root other<br/>f none /var/mail/usr/lib/nss_nis.so.1 0755 root other<br/>f none /var/mail/usr/lib/libpthread.so.1 0755 root other<br/>f none /var/mail/usr/lib/libthread.so.1 0755 root other<br/>f none /var/mail/usr/lib/nss_dns.so.1 0755 root other<br/>d none /var/mail/usr/sbin 0755 root other<br/>f none /var/mail/usr/sbin/in.pop3 0755 root other<br/>d none /var/mail/usr/local/lib 0755 root other<br/>d none /var/mail/etc 0755 root other<br/>f none /var/mail/etc/passwd 0444 root other<br/>f none /var/mail/etc/shadow 0400 root other<br/>f none /var/mail/etc/hosts.allow 0600 root other<br/>f none /var/mail/etc/nsswitch.conf 0644 root other<br/>f none /var/mail/etc/hosts.deny 0600 root other<br/>f none /var/mail/etc/netconfig 0644 root other<br/>f none /var/mail/etc/services 0444 root other<br/>f none /var/mail/etc/resolv.conf 0644 root other<br/>f none /var/mail/etc/hosts 0444 root other<br/>d none /var/mail/var 0755 root other<br/>s none /var/mail/var/mail=/<br/>d none /var/mail/dev 0755 root other<br/>c none /var/mail/dev/udp 41 0 0666 root other<br/>c none /var/mail/dev/null 13 2 0666 root other<br/>c none /var/mail/dev/conslog 21 0 0666 root other<br/>c none /var/mail/dev/log 21 5 0640 root other<br/>c none /var/mail/dev/msglog 97 1 0600 root other<br/>

Daemons I've personally chrooted without problems:

popper
rpcbind
named (It's pretty much built-in)
stunnel
Apache web server
nscd
SecurID Server
RADIUS
Squid
syslogd

Daemons that theoretically should be chrootable, though I haven't tried:

portmap
lpd (with slight creativity)
gated
routed
identd
comsat
talkd
tftpd
sadmind
Many other processes that should probably just be turned off

3.7 Logging with syslog

The syslog facility on most System V based systems uses /dev/log and
possibly other /dev/*log devices. The nice thing about this is that
you can still effectively syslog by re-creating these devices in
your chrooted environment. I've tested this on Solaris and it
worked. Non-SysV systems such as linux may not have this functionality
and will therefore need to send syslog packets to itself (machine).

Part IV: Advanced Topics

4.1 How to tell if you're chrooted

Some have asked how one can tell if you are in a chrooted environment.
Every mountpoint in a ufs filesystem has an inode number of 2. This means if
your jailed / is not on a mountpoint, the inode number will be different.
Since many systems administrators implement chroot() without considering this,
the following code could tell you if you are chrooted and not
on a mountpoint.

<br/>#include <stdio.h><br/>#include <stdlib.h><br/>#include <sys/types.h><br/>#include <sys/stat.h><br/><br/>int main(int argc, char **argv) {<br/>  struct stat x;<br/><br/>  if (stat("/", &x)) {<br/>    printf("Unable to stat /");<br/>    exit(EXIT_FAILURE);<br/>  }<br/><br/>  if (x.st_ino==2) {<br/>    printf("I am not chrooted or chrooted on a mountpoint\n");<br/>  } else {<br/>    printf("I am chrooted\n");<br/>  }<br/>  exit(EXIT_SUCCESS);<br/>}<br/>

Obviously, if you lack a complete system environment (e.g. no shadow password file, 'su' binary, or other obvious giveaways), it is easy
to tell if you are in a chroot()ed environment. Some administrators configure separate full system environments for their users, however,
in which they don't want their users to know they're chroot()ed. If the / inode is 2 and you still suspect you are in a chroot()ed
environment, you can check the following:

Check for files such as a shadow password file and 'su' binary.
Perform a process list and check for the existence of the process' binaries (some OS's allow the administrator to hide processes as well)
If your OS has a /proc, check for a /proc
Perform a 'df' and check to see if any other filesystems are mounted; if so, can you see them? if not, very suspicious.

4.2 Breaking out of a chroot()ed cell

Breaking out of a properly chrooted cell can be made quite difficult, but with the right tools is possible from within your cell. In order for someone to be
able to break out of your chrooted cell, they must be able to:

Find a vulnerability in the software you're running in the cell
Exploit this vulnerability and find a way to upload arbitrary code into memory or disk
If uploaded into memory, find a way to execute it as root
If uploaded to disk, also upload a shell into your cell and call the shell to execute the code as root

Uploading and executing the arbitrary code into memory can be as simple as
uploading and executing root shell code as done in many exploits. The complete
code to break out of a chroot()ed cell is only a mere 139 bytes! Once they've accomplished this, breaking out of the cell is relatively simple. Notice, however, that the above scenario is only possible if the software you are running is running as root. It is strongly recommended, to make chroot more effective, that you run your system software as a non-privileged user. This makes it much
more difficult to break out of a cell.

Breaking out is beyond the scope of this document, but the following link is a pretty good explanation of one method of doing it: http://www.bpfh.net/computing/docs/chroot-break.html With careful planning and
understanding of chroot()'s vulnerabilities, it should be possible to make breaking out extremely difficult. There are
also some great tools available for intercepting system calls (such as root shell calls) to harden your OS.

4.3 Frequent chroot() Mistakes

Even in a chroot()ed environment, running the target software with certain loopholes can seriously degrade the security that chroot() provides. The following is a growing list of common mistakes many administrators can make when chroot()ing processes:

Mistake: Running chroot()ed processes with root privileges

Severity: Critical

One of the most common misperceptions is that running your software as root in a chroot()ed environment is safe. This is completely untrue.
The idea behind chroot() is to protect your system from someone exploiting a vulnerability in your target software. If your software
indeed does have a vulnerability, running the software as root provides many ways for your system to be compromised. Sure, chroot() may
protect you from the average script kiddie, but an experienced hacker can easily break out of the cell (if they have root privileges),
play with your chroot()ed /dev devices, kill, trace, or snoop on other processes, and wreak general havoc all from within their cell if
the target software is running as root.

Solution: Modify your implementation to change its user and group id's after chroot()ing. This can be done either by manually
modifying the source code to perform setuid() and setgid() function calls, or by writing a small wrapper to do it. Since this software is
running in a chroot()ed environment, you should be able to safely change file permissions around to accomodate a non-root user. For example,
if you are chroot()ing popper, you can modify the popper to run as nobody, but in the group owning all mailboxes. This will enable you
to service mail requests without being super user.

Mistake: Using a copy of the shadowed password file

Severity: High

In our popper example, we copied the shadow password file into the chroot()ed directory to provide authentication. Unfortunately, if
there is a vulnerability in the popper, it may allow a hacker to steal the shadow password file, run a cracker, and gain access to the system through the front door.

Solution: There are a few different solutions to this. First, if you are able to implement an external method of authentication (for example RADIUS, SecurID, or even utilizing the PAM libraries) that will allow you to authenticate without having a copy of the shadow file inside the jail, this is the best solution. If this is impractical for your implementation, another way to handle this would be to parse the shadow password file prior to copy and remove any system or privileged (shell) accounts from the file. If some privileged accounts must be used,
set up your shadow file to use different passwords for email than used for the system.

Mistake: Creating virtual system environments off of a mountpoint

Severity: Low

By chroot()ing a directory that is not also a mountpoint, you provide a small hole for detecting that the user is
chroot()ed. As we discussed earlier, all mountpoints have an inode number of 2. By chroot()ing off of a mountpoint, you advertise that
the process is chroot()ed. This is not very dangerous as it doesn't provide a way to break out of the chroot() but is nevertheless providing
some information the hacker shouldn't have. If you have a relatively small chroot()ed environment where the lack of several system files
makes it obvious that you are chroot()ed, there is no reason to correct this, but if you are trying to emulate a complete environment
(e.g. virtual machines), it may be a good idea to put your jail on a mountpoint.

Solution: If you are trying to emulate a full system environment (e.g. virtual machines) and do not want your users to know they're
chroot()ed, place the jail on a mountpoint.

Article from jonathan@networkdweebs.comjonathan@networkdweebs.com

2.2 Introduction to truss, lsof, and ldd

truss (or strace)

If you 'man truss' you'll see that truss is a tool designed to trace system
calls and signals. strace is a similar tool used in Linux.
System calls include calls to libraries and accessing
files. Truss is a great way to find out what resources a particular daemon
is using. To use truss, you simply type 'truss' followed by the command
you would normally use to start the daemon. Output you are looking for will
look like this:

open("/usr/lib/libc.so.1", O_RDONLY) = 3

Program opened the libc library

open64("/usr/local/etc/my.config", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3

Program opened /usr/local/etc/my.config for writing

Running truss on a daemon should immediately give you a long list of files to
start with on your search. It's a good idea to utilize all the different
features and commands of a daemon and watch for new files in truss. Without
truss, you'll end up breaking your chrooted jail once for every time you're
missing a data file.

lsof

lsof is another great tool for finding the resources a daemon uses. lsof
is designed to operate on daemons that are already running. It will show you
open files, network connections, and much more. lsof does not come with
most systems, as it is a third party tool. It can be downloaded from many
sites such as www.sunfreeware.com. Sources are readily available via
anonymous ftp from ftp://vic.cc.purdue.edu/pub/tools/unix/lsof.

ldd

ldd is a tool handy for determing what library resources your program will
need in order to execute. ldd stands for 'list dynamic dependencies'. Running
ldd followed by the path and filename of the daemon you are chrooting will give
you a list of libraries you must plan to copy over to the jail.

A note on dynamic libraries: If you have access to the
sources for the program you are chrooting, you may wish to compile it
with static libraries. This will prevent you from having to copy over
libraries. If you use an operating system whose manufacturer updates
the libraries frequently, however, you may wish to remain dynamic.

2.3 Determining datafile dependencies

The first little exercise we went through in 2.1 got us thinking about what
datafiles the daemon is dependent on. Some are relocatable, while others
we have to build a jail around. Only two of the three examples from the
previous section were viable (Eudora Qpopper and Sun's RPCBIND). Both are
very good candidates for chrooting and both have had severe security
vulnerabilities in the past. Now lets move on to mapping out the actual
datafiles that we're going to need to move or build around for these two
examples. It's perfectly alright to miss a few small files as we can always
find a solution for them before we turn the new service up. There are several
ways to determine datafiles used. These include, but are not limited to:

The obvious (duh, a mail popper accesses mail folders)
Examining configuration files
Using tools such as 'lsof' and 'truss'
(Preferably not) Breaking the daemon later on in its jail

<br/>hosts.allow<br/>hosts.deny<br/>netconfig<br/>nsswitch.conf<br/>passwd<br/>resolv.conf<br/>services<br/>shadow<br/>

Sun RPCBIND

The following files from /etc (copied in periodically)

<br/>hosts<br/>netconfig<br/>nsswitch.conf<br/>resolv.conf<br/>services<br/>

<br/>/usr/lib/libnsl.so.1<br/>/usr/lib/libsocket.so.1<br/>/usr/lib/libresolv.so.2<br/>/usr/lib/libmail.so.1<br/>/usr/lib/librt.so.1<br/>/usr/lib/libcrypt_i.so.1<br/>/usr/lib/libc.so.1<br/>/usr/lib/libdl.so.1<br/>/usr/lib/libmp.so.2<br/>/usr/lib/libaio.so.1<br/>/usr/lib/libgen.so.1<br/>/usr/lib/libpthread.so.1<br/>/usr/lib/libthread.so.1<br/>

Sun RPCBIND

<br/>/usr/lib/libsocket.so.1<br/>/usr/lib/libnsl.so.1<br/>/usr/lib/libdl.so.1<br/>/usr/lib/libc.so.1<br/>/usr/lib/libmp.so.2<br/>

<br/>mkdir /var/rpcbind<br/>mkdir /var/rpcbind/dev<br/>mkdir /var/rpcbind/etc<br/>mkdir /var/rpcbind/tmp<br/>mkdir -p /var/rpcbind/usr/lib<br/>

<br/>mkdir /var/mail/dev<br/>mkdir /var/mail/etc<br/>mkdir -p /var/mail/usr/lib <br/>mkdir -p /var/mail/usr/local/lib<br/>mkdir -p /var/mail/usr/sbin<br/>mkdir -p /var/mail/usr/share/lib/zoneinfo/US<br/>

<br/>mkdir /var/mail/var<br/>ln -s / /var/mail/var/mail <br/>

Now when our chrooted popper goes to access /var/mail, it'll be pointed to /, which is the real
/var/mail!

Our final products should look like:

Sun RPCBIND

<br/><br/>drwxr-xr-x   7 root     other        512 Aug  1 18:31 ./<br/>drwxr-xr-x  34 root     sys          512 Aug  1 18:07 ../<br/>drwxr-xr-x   2 root     other        512 Aug  1 18:16 dev/<br/>drwxr-xr-x   2 root     other        512 Aug  1 18:10 etc/<br/>drwxr-xr-x   2 root     other        512 Aug  1 18:31 tmp/<br/>drwxr-xr-x   6 root     other        512 Jul 28 15:55 usr/<br/>drwxr-xr-x   3 root     other        512 Aug  1 18:19 var/<br/>

Eudora Qpopper

<br/><br/>drwxrwxrwt   7 root     mail         512 Oct 18 19:36 ./<br/>drwxr-xr-x  34 root     sys          512 Aug  1 18:07 ../<br/>drwxr-xr-x   2 root     other        512 Jul 29 13:33 dev/<br/>drwxr-xr-x   2 root     other        512 Jul 28 16:18 etc/<br/><br/>            - Some mailboxes mixed in here -<br/><br/>drwxr-xr-x   6 root     other        512 Jul 28 15:55 usr/<br/>drwxr-xr-x   2 root     other        512 Aug  1 18:06 var/<br/>

<br/><br/>% ls -l /var/rpcbind/etc<br/>-r--r--r--   1 root     other         90 Jul 28 16:18 hosts<br/>-rw-r--r--   1 root     other       1239 Jul 28 16:09 netconfig<br/>-rw-r--r--   1 root     other        835 Jul 28 15:32 nsswitch.conf<br/>-rw-r--r--   1 root     other        140 Jul 28 16:14 resolv.conf<br/>-r--r--r--   1 root     other       3649 Jul 28 16:14 services<br/>

Eudora Qpopper

<br/><br/>% ls -l /var/mail/etc<br/><br/>-r--r--r--   1 root     other         90 Jul 28 16:18 hosts<br/>-rw-------   1 root     other         73 Oct 18 19:15 hosts.allow<br/>-rw-------   1 root     other          9 Jul 28 15:58 hosts.deny<br/>-rw-r--r--   1 root     other       1239 Jul 28 16:09 netconfig<br/>-rw-r--r--   1 root     other        835 Jul 28 15:32 nsswitch.conf<br/>-r--r--r--   1 root     other        815 Oct 18 19:15 passwd<br/>-rw-r--r--   1 root     other        140 Jul 28 16:14 resolv.conf<br/>-r--r--r--   1 root     other       3649 Jul 28 16:14 services<br/>-r--------   1 root     other        502 Oct 18 19:15 shadow<br/>

The cron jobs we came up with look like this. Whatever works for you...

<br/>0,30 * * * * cp -p /etc/passwd /var/mail/etc/passwd<br/>0,30 * * * * cp -p /etc/shadow /var/mail/etc/shadow<br/>0 0 * * * cp -p /etc/hosts.* /var/mail/etc/<br/>0 0 0 * * cp -p /etc/services /var/rpcbind/etc<br/>0 0 0 * * cp -p /etc/resolv.conf /var/mail/etc<br/>

<br/>% ls -l /var/rpcbind/usr/lib<br/><br/>-rwxr-xr-x   1 root     other     200292 Jul 28 15:27 ld.so.1*<br/>-rwxr-xr-x   1 root     other      41628 Jul 28 15:28 libaio.so.1*<br/>-rwxr-xr-x   1 root     other     938940 Jul 28 15:28 libc.so.1*<br/>-rwxr-xr-x   1 root     other      15616 Jul 28 15:27 libcrypt_i.so.1*<br/>-rwxr-xr-x   1 root     other       4448 Jul 28 15:28 libdl.so.1*<br/>-rwxr-xr-x   1 root     other      40540 Jul 28 15:28 libgen.so.1*<br/>-rwxr-xr-x   1 root     other      29548 Jul 28 15:27 libmail.so.1*<br/>-rwxr-xr-x   1 root     other      19584 Jul 28 15:28 libmp.so.2*<br/>-rwxr-xr-x   1 root     other     730672 Jul 28 15:27 libnsl.so.1*<br/>-rwxr-xr-x   1 root     other      35308 Jul 28 15:57 libpthread.so.1*<br/>-rwxr-xr-x   1 root     other     326336 Jul 28 15:27 libresolv.so.2*<br/>-rwxr-xr-x   1 root     other      39048 Jul 28 15:27 librt.so.1*<br/>-rwxr-xr-x   1 root     other      65876 Jul 28 15:27 libsocket.so.1*<br/>-rwxr-xr-x   1 root     other     166624 Jul 28 15:58 libthread.so.1*<br/>-rwxr-xr-x   1 root     other      19648 Jul 28 16:17 nss_dns.so.1*<br/>-rwxr-xr-x   1 root     other      38832 Jul 28 15:33 nss_files.so.1*<br/>-rwxr-xr-x   1 root     other      38292 Jul 28 15:33 nss_nis.so.1*<br/>-rwxr-xr-x   1 root     other      12284 Aug  1 18:15 straddr.so*<br/><br/>% ls -l /var/mail/usr/lib /var/mail/usr/local/lib<br/><br/>-rwxr-xr-x   1 root     other     200292 Jul 28 15:27 ld.so.1*<br/>-rwxr-xr-x   1 root     other      41628 Jul 28 15:28 libaio.so.1*<br/>-rwxr-xr-x   1 root     other     938940 Jul 28 15:28 libc.so.1*<br/>-rwxr-xr-x   1 root     other      15616 Jul 28 15:27 libcrypt_i.so.1*<br/>-rwxr-xr-x   1 root     other       4448 Jul 28 15:28 libdl.so.1*<br/>-rwxr-xr-x   1 root     other      40540 Jul 28 15:28 libgen.so.1*<br/>-rwxr-xr-x   1 root     other      29548 Jul 28 15:27 libmail.so.1*<br/>-rwxr-xr-x   1 root     other      19584 Jul 28 15:28 libmp.so.2*<br/>-rwxr-xr-x   1 root     other     730672 Jul 28 15:27 libnsl.so.1*<br/>-rwxr-xr-x   1 root     other      35308 Jul 28 15:57 libpthread.so.1*<br/>-rwxr-xr-x   1 root     other     326336 Jul 28 15:27 libresolv.so.2*<br/>-rwxr-xr-x   1 root     other      39048 Jul 28 15:27 librt.so.1*<br/>-rwxr-xr-x   1 root     other      65876 Jul 28 15:27 libsocket.so.1*<br/>-rwxr-xr-x   1 root     other     166624 Jul 28 15:58 libthread.so.1*<br/>-rwxr-xr-x   1 root     other      19648 Jul 28 16:17 nss_dns