System Calls and the Trap Mechanism

The system-call interface

A system call provides the controlled gateway from user space into the kernel. Each call is identified by a number (e.g., on x86-64 Linux: write = 1, open = 2, close = 3, fork = 57). The user program puts the system-call number and arguments in designated registers, then invokes the trap instruction.

The trap flow

1. User space preparation: The C library (libc) wraps the system call in a function. The wrapper places the syscall number in rax (x86-64) or x8 (ARM64), places up to six arguments in registers (rdi, rsi, rdx, r10, r8, r9 on x86-64), and issues the trap instruction (syscall).

2. Hardware trap: The CPU atomically:

   • Switches the privilege level from user to kernel.
   • Saves the current program counter and stack pointer into kernel-controlled registers.
   • Jumps to a kernel entry point defined at boot time (e.g., entry_SYSCALL_64 in Linux).
   • Switches to the kernel stack.

3. Kernel dispatching: The kernel entry code saves all general-purpose registers onto the kernel stack (preserving the user context), then uses the syscall number in rax to index into the system-call table. Each entry is a function pointer. The kernel calls the appropriate handler (e.g., __x64_sys_write).

4. Execution: The handler validates arguments (checking pointers belong to the calling process and do not point into kernel memory), then performs the operation.

5. Return: The kernel restores user registers from the saved state on the kernel stack, executes a return-from-interrupt instruction (sysret on x86-64, eret on ARM64), and the CPU atomically switches back to user mode.

Cost of a system call

A system call is orders of magnitude more expensive than a user-space function call because:

• The trap instruction flushes the pipeline and incurs a branch-prediction penalty.
• The kernel must save and restore dozens of registers.
• Cache state is disturbed — kernel code and data displace user cache lines.
• Argument validation adds overhead (especially for pointer-heavy calls).
• On return, TLB entries may need to be reloaded.

Historically, early Linux system calls cost hundreds of cycles; modern mitigations (e.g., KPTI for Meltdown) have made them more expensive again, sometimes exceeding a thousand cycles.

Classes of system calls

Category	Examples
Process control	fork, execve, exit, waitpid, kill
File management	open, read, write, close, lseek, stat, unlink
Device management	ioctl, mmap (for device memory)
Information	getpid, gettimeofday, sysinfo
Communication	pipe, socket, connect, send, recv, shmget
Memory management	mmap, munmap, brk, sbrk

Summary

• System calls are numbered, dispatched through a trap-instruction → kernel-entry → syscall-table path.
• The trap instruction is the only way to cross from user into kernel mode.
• Syscall overhead is significant (hundreds to thousands of cycles) but unavoidable.
• The kernel must validate every argument to prevent user code from corrupting or reading kernel memory.