We currently migrate to MediaWiki from our old installation, but not all content has been migrated yet. Take a look at the Wiki Team page for instructions how to help or browse through our new wiki at wiki.linux-vserver.org to find the information already migrated.

Goal

Setup a secured chroot/jailed hosting environment using a minimal debootstrapped testing vserver as a reference server to unify against and a host machine with grsecurity resulting in a copletely isolated virtual hosting environment.

What this does not detail

This document only shows you how to setup unified vserver chroot services, not entire virtual machines (ala UML). The goal is to not have a complete linux environment, but only those things which are needed in order to run a particular service.

Benefits

  • Chroot fixes
There are two ways one may escape from a chroot environment. One is to setup a block device special file, open it with some user-land file system browser. At this point, you are bypassing both the chroot restriction and all file system access control.
This does not work on a vserver because the special device are not created in /dev and the CAP_SYS_MKNOD capability is disabled. Even root can't create special files in a vserver.
The second is changing the root while keeping the current directory behind.
This is a trick to exploit a flaw in the chroot() system call. The system call is changing the logical root directory for the calling process but does not change the current working directory of the process. So the current working directory is (generally) left behind the new root directory. The /usr/sbin/chroot command takes care of this flaw by changing the working directory, but the system call does not do that. So if you process has now a working directory behind (out of scope) the new root, it is allow to change it up to the real root by doing multiple chdir ("..") system call. The process kind of escapes from the kernel radar.
  • Complete isolation
When attackers are allowed only minimum access to software they can compromise those systems without compromising things for everybody.
  • Limited resource overhead
A vserver does not use any resource by itself. There is no "invisible" overhead for each vserver. The overhead comes from the tasks you are running inside the vserver. You can expect the exact same performance in a vserver as compared to the root server. There is no overhead. Processes running in the vserver are talking directly to the kernel. Only few system calls (kill for one) have special checks to insure processes isolation.
In addition, using this setup libraries and binaries are read-only and shared in a "unified" environment. This allows us to have multiple instances of libraries and binaries installed with little disk space and memory footprint used. For example, a glibc using up 5 megabytes used in 10 vservers only contributes .5MB to each, and reduces the memory footprint by a factor of 10.

The future

The future vserver kernel patch and utilities make some of these steps unnecessary. Specifically unification is apparantly much easier, and the hereditary +t is gone, among other things.

Prepare the host system (this is only done once)

1.download latest kernel source from ftp.kernel.org

# cd /usr/src
# wget ftp://ftp.kernel.org/pub/linux/kernel/v2.4/linux-2.4.25.tar.bz2

2. get the grsec enabled patch and uncompress it

# wget http://www.sandino.net/parches/vserver/linux-2.4.25-grsec-1.9.14-vserver-1.26.patch.gz
# gunzip linux-2.4.25-grsec-1.9.14-vserver-1.26.patch.gz

3. Uncompress, untar kernel

# cd /usr/src
# tar -jxf linux-2.4.25
# cd patches

4. Patch the kernel

# cd /usr/src/linux-2.4.25
# patch -p1 < ../patches/linux-2.4.25-grsec-1.9.14-vserver-1.26.patch

5. Configure, compile, and install the kernel, make sure you enable:

# make menuconfig
# make dep; make clean; make bzImage

6. Install the vserver tools

# wget
http://www.13thfloor.at/vserver/s_release/v1.26/util-vserver-0.28.tar.bz2
# tar -jxvf util-vserver-0.28.tar.bz2
# ./configure; make; make install

7. Set up the general vserver directories and the barrier and the

/proc security

# mkdir /vservers
# chmod 000 /vservers
# chattr +t /vservers

This causes some problems because the current stable version of the vserver kernel stuff has +t as a hereditary attribute, so any directory or file created underneath /vservers will also have +t, this is *NOT* desirable. Only files which are unified should have +t, but not the directories themselves. We solve this with our scripts that we use to create our vservers and unifications below. It is also solved in the alpha version of the vserver kernel patch.

Prepare the reference server (this is done once, but needs to be maintained)

8. Create a reference vserver installation, called zunzuncito (smallest bird

in the world Image41.jpg), using the script provided [here]

Note, I hackishly modified this to work with testing/sarge, which required that we install the testing version of debootstrap and did not go smoothly, I had to massage things by hand to get it to work. I wanted to create a testing vserver as our reference server so that our vservers could have testing packages, but the host server remains the stable woody.

# /home/micah/debian-newvserver.sh --hostname zunzuncito --domain domain --ip 192.168.0.100

9. Enter the reference server and install a package (networking has to work):
<code>
# vserver zunzuncito enter
# apt-get install apache apache-common

If you get errors like: "Can't set the new security context: Invalid argument" then the vserver is not running. First start the vserver with:

# vserver zunzuncito start

Create a unified vserver instance (this is done many times)

15. Gather the files associated with each package (this requires the debian package apt-rdepends, also this script is located and run within zunzuncito /root/rdepends):

#!/bin/sh

packages="apache"

for i in $packages; do
  dpkg -L $i > /tmp/files
  for package in apt-rdepends $i |grep Depends: | awk '{print $2}'
|sort -u;
  do
        dpkg -L $package 2>&1 | \
        egrep -v "not found|not installed|to examine archive files" | \
        egrep -v "to list their
contents|/usr/share/doc|/usr/share/man" \
        >> /tmp/files;
  done
done

sort -u /tmp/files > /tmp/files2
rm /tmp/files
mv /tmp/files2 /tmp/files

16. To unify, create hard links from /vserver/zunzuncito/$files to /vserver/unified1/$files (make sure directories exist first), and then set the immutability and immutable link invert attributes. This script is on the host server (in our case, raven) in /root/link:

#!/bin/sh

reference=zunzuncito
vserver=$1

if [[ ! -d $vserver ]]
then
        mkdir $vserver
fi

cd /vservers/$reference

for file in cat /vservers/$reference/tmp/files
do
        echo ${file:1} | cpio -v --link -d -m -p /vservers/$vserver
done

find /vservers/$vserver -type f | xargs chattr +it
find /vservers/$vserver -type d | xargs chattr -t

echo ""
echo "Dont forget to create /etc/vservers/$vserver.conf"

17. Create any additional required things that are needed for your package to run in this chrooted setup (we are creating templates for these so that they are done easier), this example is for getting

apache to work using the debian apache package:

create /etc/apache/httpd.conf /etc/apache/mime.types /etc/apache/modules.conf
(make sure the User and Group are set in httpd.conf as what you set
below)

mkdir -p /var/run
chattr -t /var/run

echo 'www-data:x:888:888:Web Account:/var/www:/usr/bin/bah' >  etc/passwd
chattr -t /etc/passwd
echo 'www-data:x:888:' >  etc/group
chattr -t /etc/group

cat > /etc/nsswitch.conf
passwd: files
shadow: files
group: files
hosts: files dns
EOF
chattr -t /etc/nsswitch.conf

cat > /etc/resolv.conf
domain mynet.home
## use the IP address of your naming server
## if bind is not installed on your web server
#nameserver 192.168.196.xxx
## use this if your web server is a (caching) name server
nameserver 127.0.0.1
chattr -t /etc/hosts

echo 'www-data:*:10882:-1:99999:-1:-1:-1:134537804' >  etc/shadow
chmod 400 etc/shadow
chattr -t /etc/shadow

cat > /etc/hosts
127.0.0.1 localhost loopback
192.168.0.100 ns.mynet.home ns
chattr -t /etc/hosts

mkdir /var/www
chattr -t /var/www
make sure the permissions on /var are correct: chmod 755 /var

18. Start up apache in the vserver

# vserver $vserver exec /usr/sbin/httpd

19. vserver command notes

# vserver-stat
# vserver zunzuncito enter (with a shell)
# vps
# vtop
# vpstree

These are simply wrappers which are actually doing:

# chcontext --ctx 1 pstree $*
eg. chcontext --ctx <CID> ps auxw

Upgrading packages (to be written)

Process for upgrading:

a) figure out what files won't be needed after the update

b) remove those files and the files to be installed from the unified children

c) chattr +i +t the new files installed in the reference server

d) hardlink the new files into the children

Todo

write templates for php4, ssh, ftp, etc.

install baobob for account requests/automatic creation of these vservers

figure out networking :p

Reference material

thanks to Bertl!