![[Home]](http://wiki.linux-vserver.org/skins/common/images/wiki.png)
We currently migrate to MediaWiki from our old installation, but not all content has been migrated yet. Take a look at the Wiki Team page for instructions how to help or browse through our new wiki at wiki.linux-vserver.org to find the information already migrated.
While this chapter is mainly of interest to kernel developers it might be fun to take a small peek behind the curtain to get a glimpse how everything really works.
For a long time Linux-VServer used a few different Syscalls to accomplish different aspects of the work, but very soon the number of required commands grew large, and the Syscalls started to have magic values, selecting the desired behavior.
Not too long ago, a single syscall was reserved for Linux-VServer, and while the opinion on that might differ from developer to developer, it was generally considered a good decision not to have more than one syscall.
The advantage of different Syscalls would be simpler handling of the Syscalls on different architectures; however, this hasn't been a problem so far, as the data passed to and from the kernel has strong typed fields conforming to the C99 types.
Regardless, the availability of one system call required the creation of a multiplexor, which decides, based on some selector, what specific command is to be executed, and then passes on the remaining arguments to that command, which does the actual work.
extern asmlinkage long sys_vserver(uint32_t cmd, uint32_t id, void __user *data)
The Linux-VServer syscall is passed three arguments regardless of what actual command is specified: a command (cmd), a number (id), and a user-space data-structure of yet unknown size.
To allow for some structure for debugging purposes and some kind of command versioning, the cmd is split into three parts: the lower 12 bit contain a version number, then 4 bits are reserved, the upper 16 bits are divided into 8 bit command and 6 bit category, again reserving 2 bits for the future.
There are 64 Categories with up to 256 commands in each category, allowing for 4096 revisions of each command, which is far more than will ever be required.
Here is an overview of the categories already defined, and their numerical value:
Syscall Matrix V2.6
|VERSION|CREATE |MODIFY |MIGRATE|CONTROL|EXPERIM| |SPECIAL|SPECIAL|
|STATS |DESTROY|ALTER |CHANGE |LIMIT |TEST | | | |
|INFO |SETUP | |MOVE | | | | | |
-------+-------+-------+-------+-------+-------+-------+ +-------+-------+
SYSTEM |VERSION|VSETUP |VHOST | | | | |DEVICES| |
HOST | 00| 01| 02| 03| 04| 05| | 06| 07|
-------+-------+-------+-------+-------+-------+-------+ +-------+-------+
CPU | |VPROC |PROCALT|PROCMIG|PROCTRL| | |SCHED. | |
PROCESS| 08| 09| 10| 11| 12| 13| | 14| 15|
-------+-------+-------+-------+-------+-------+-------+ +-------+-------+
MEMORY | | | | | | | |SWAP | |
| 16| 17| 18| 19| 20| 21| | 22| 23|
-------+-------+-------+-------+-------+-------+-------+ +-------+-------+
NETWORK| |VNET |NETALT |NETMIG |NETCTL | | |SERIAL | |
| 24| 25| 26| 27| 28| 29| | 30| 31|
-------+-------+-------+-------+-------+-------+-------+ +-------+-------+
DISK | | | | | | | |INODE | |
VFS | 32| 33| 34| 35| 36| 37| | 38| 39|
-------+-------+-------+-------+-------+-------+-------+ +-------+-------+
OTHER | | | | | | | |VINFO | |
| 40| 41| 42| 43| 44| 45| | 46| 47|
=======+=======+=======+=======+=======+=======+=======+ +=======+=======+
SPECIAL| | | | |FLAGS | | | | |
| 48| 49| 50| 51| 52| 53| | 54| 55|
-------+-------+-------+-------+-------+-------+-------+ +-------+-------+
SPECIAL| | | | |RLIMIT |SYSCALL| | |COMPAT |
| 56| 57| 58| 59| 60|TEST 61| | 62| 63|
-------+-------+-------+-------+-------+-------+-------+ +-------+-------+
The definition of those Commands is simplified by some macros, so for example the commands to get and set the Context Flags are defined like this:
#define VCMD_get_cflags VC_CMD(FLAGS, 1, 0) #define VCMD_set_cflags VC_CMD(FLAGS, 2, 0) extern int vc_get_cflags(uint32_t, void __user *); extern int vc_set_cflags(uint32_t, void __user *);
Note that the command itself is not passed to the actual command implementation, only the id and the pointer to user-space data.
There are many different data structures used by different parts of the implementation; while only a few examples are given here, all utilized structures can be found in the source.
The Context Data Structure consists of a few fields required to manage the contexts, and handle context destruction, as well as future hierarchical contexts.
Logically separated sections of that structure, like for the scheduler or the context limits are defined in separate structures, and incorporated into the main one.
struct vx_info {
struct list_head vx_list; /* linked list of contexts */
xid_t vx_id; /* context id */
atomic_t vx_refcount; /* refcount */
struct vx_info *vx_parent; /* parent context */
struct namespace *vx_namespace; /* private namespace */
struct fs_struct *vx_fs; /* private namespace fs */
uint64_t vx_flags; /* context flags */
uint64_t vx_bcaps; /* bounding caps (system) */
uint64_t vx_ccaps; /* context caps (vserver) */
pid_t vx_initpid; /* PID of fake init process */
struct _vx_limit limit; /* vserver limits */
struct _vx_sched sched; /* vserver scheduler */
struct _vx_cvirt cvirt; /* virtual/bias stuff */
struct _vx_cacct cacct; /* context accounting */
char vx_name[65]; /* vserver name */
};
Here as example the Scheduler Substructure:
struct _vx_sched {
spinlock_t tokens_lock; /* lock for this structure */
int fill_rate; /* Fill rate: add X tokens ... */
int interval; /* Divisor: ... each Y jiffies */
atomic_t tokens; /* current number of tokens */
int tokens_min; /* Limit: minimum for unhold */
int tokens_max; /* Limit: no more than N tokens */
uint32_t jiffies; /* bias: integral multiple of Y */
uint64_t ticks; /* token tick events */
cpumask_t cpus_allowed; /* cpu mask for context */
};
The main idea behind this separation is that each substructure belongs to a logically distinct part of the implementation which provides an init and cleanup function for this structure, thus simplifying maintainability and readability of those structures.
As an example for the data structure used to control a specific part of the context from user-space, here is a scheduler command and the utilized data structure to set the properties:
#define VCMD_set_sched VC_CMD(SCHED, 1, 2)
struct vcmd_set_sched_v2 {
int32_t fill_rate; /* Fill rate: add X tokens ... */
int32_t interval; /* Divisor: ... each Y jiffies */
int32_t tokens; /* current number of tokens */
int32_t tokens_min; /* Limit: minimum for unhold */
int32_t tokens_max; /* Limit: no more than N tokens */
uint64_t cpu_mask; /* Mask: allowed cpus */
};
Basically all the accounting and limit stuff are defined as macros or inline functions capable of handling the different resources, hiding the underlying implementation wherever possible.
#define vx_acc_sock(v,f,p,s) \
__vx_acc_sock((v), (f), (p), (s), __FILE__, __LINE__)
static inline void __vx_acc_sock(struct vx_info *vxi,
int family, int pos, int size, char *file, int line)
{
if (vxi) {
int type = vx_sock_type(family);
atomic_inc(&vxi->cacct.sock[type][pos].count);
atomic_add(size, &vxi->cacct.sock[type][pos].total);
}
}
#define vx_sock_recv(sk,s) \
vx_acc_sock((sk)->sk_vx_info, (sk)->sk_family, 0, (s))
#define vx_sock_send(sk,s) \
vx_acc_sock((sk)->sk_vx_info, (sk)->sk_family, 1, (s))
#define vx_sock_fail(sk,s) \
vx_acc_sock((sk)->sk_vx_info, (sk)->sk_family, 2, (s))
And this general definition is then used where appropriate, for example in the __sock_sendmsg() function like this:
len = sock->ops->sendmsg(iocb, sock, msg, size);
if (sock->sk) {
if (len == size)
vx_sock_send(sock->sk, size);
else
vx_sock_fail(sock->sk, size);
}
#define vx_pages_avail(m, p, r) \
__vx_pages_avail((m)->mm_vx_info, (r), (p), __FILE__, __LINE__)
static inline int __vx_pages_avail(struct vx_info *vxi,
int res, int pages, char *file, int line)
{
if (!vxi)
return 1;
if (vxi->limit.rlim[res] == RLIM_INFINITY)
return 1;
if (atomic_read(&vxi->limit.res[res]) +
pages < vxi->limit.rlim[res])
return 1;
return 0;
}
#define vx_vmpages_avail(m,p) vx_pages_avail(m, p, RLIMIT_AS)
#define vx_vmlocked_avail(m,p) vx_pages_avail(m, p, RLIMIT_MEMLOCK)
#define vx_rsspages_avail(m,p) vx_pages_avail(m, p, RLIMIT_RSS)
And again the test against those limits at certain places, for example here in copy_process()
/* check vserver memory */
if (p->mm && !(clone_flags & CLONE_VM)) {
if (vx_vmpages_avail(p->mm, p->mm->total_vm))
vx_pages_add(p->mm->mm_vx_info,
RLIMIT_AS, p->mm->total_vm);
else
goto bad_fork_free;
}
void vx_vsi_uptime(struct timespec *uptime)
{
struct vx_info *vxi = current->vx_info;
set_normalized_timespec(uptime,
uptime->tv_sec - vxi->cvirt.bias_tp.tv_sec,
uptime->tv_nsec - vxi->cvirt.bias_tp.tv_nsec);
return;
}
if (vx_flags(VXF_VIRT_UPTIME, 0))
vx_vsi_uptime(&uptime, &idle);
<a href="index.php?action=edit&page=Linux-VServer-Paper-11">Go Back </a>