I/O & Syscalls Explained

The filesystems post ended with an open thread: files live on disk, but programs read keyboards, screens, and sockets through the same syscalls. The hardware I/O post showed the device side (controllers, interrupts, DMA); this post shows the OS side. We will look at how syscalls cross into the kernel in practice, what blocking really means, the non-blocking and multiplexed alternatives, the layers of buffering along the way, and the full path of one read from program to platter and back.

Syscalls in practice: the cost of crossing

A system call crosses privilege levels: trap instruction, argument validation, mode switch, work, copy results back, return. The crossing costs hundreds of nanoseconds to microseconds, trivial once, significant a million times per second. Three techniques keep the bill down.

First, batching: the C library buffers. printf accumulates in userspace and issues one write for thousands of characters; malloc subdivides kernel pages so most allocations never trap. Second, vector I/O (readv, writev): one crossing moves many scattered buffers. Third, avoidance: vDSO maps a few read-only kernel results (time, CPU number) into userspace so gettimeofday often never traps at all.

program              libc                  kernel
printf("hi") ──▶  buffer it          ──▶  (nothing yet)
printf("ho") ──▶  buffer it          ──▶  (nothing yet)
fflush()     ──▶  one write(fd,"hiho")──▶  validate, copy, queue to device

Strace from the overview post shows the seam: every line is one crossing, with arguments, return value, and time spent. Reading strace output is reading the program’s conversation with the OS, syscall by syscall.

Blocking I/O: sleep until ready

Blocking I/O¹ is the default: a read on an empty socket suspends the thread until data arrives. The kernel parks the thread in the blocked state, schedules someone else, and wakes the sleeper when the device interrupts or the data lands. The thread pays nothing while waiting; the CPU stays busy elsewhere. Simple to program (code reads top to bottom), efficient under light concurrency, awkward at scale: ten thousand connections need ten thousand threads, each with its own stack and scheduling cost.

thread calls read(fd) ──▶ no data → BLOCKED, scheduler runs others
device interrupts ──▶ kernel copies data ──▶ thread READY → RUNNING, read returns

Blocking is honest waiting: the thread declared it cannot proceed, and the kernel believed it.

Non-blocking and multiplexed I/O: ask, don’t sleep

Non-blocking I/O flips the contract: the call returns immediately, with data or with EAGAIN (“not ready, try later”). The thread never sleeps, but somebody must retry, which naive polling turns into a CPU-burning loop. The answer is I/O multiplexing²: one syscall watches many descriptors and sleeps until at least one is ready.

// wait until some fds are readable, then handle only those
struct pollfd fds[] = {{fd1, POLLIN}, {fd2, POLLIN}};
poll(fds, 2, -1);
for (int i = 0; i < 2; i++)
    if (fds[i].revents & POLLIN)
        handle(fds[i].fd);

select and poll scan the whole set per call; epoll (Linux) and kqueue (BSD/macOS) keep the interest set in the kernel and report only changes, scaling to tens of thousands of connections. This is the engine inside nginx, Node.js, and every event loop: one thread, many sockets, wakeups only where work waits. Asynchronous I/O (io_uring on Linux) pushes further: submit operations through shared queues and harvest completions, batching submissions and avoiding per-operation syscalls entirely.

Buffers everywhere: the data sits still so it can move fast

Between program and device sit layers of staging memory, each hiding a speed mismatch:

• Userspace buffers (stdio, language runtimes): coalesce tiny writes into few syscalls.
• Page cache: the kernel keeps file data in RAM; reads hit cache instead of disk, writes land in cache and flush lazily (write-back). Most “disk” I/O never reaches any disk.
• Device queues: the I/O scheduler orders pending requests (merge adjacent sectors, prioritize deadlines, keep the device streaming).

printf ──▶ stdio buffer ──▶ write() ──▶ page cache ──▶ I/O scheduler ──▶ device
           (userspace)        (kernel memory)            (queue)          (DMA)

Each layer answers the same question with more patience than the one above: wait a little, batch a lot, touch slow hardware rarely.

One read, end to end

Trace read(fd, buf, 4096) on a disk file, cache cold:

1. trap: validate fd, check permissions, find file offset
2. page cache lookup: miss → allocate page, queue device read
3. driver programs DMA: device copies blocks straight to RAM, CPU free
4. interrupt: transfer done → wake blocked thread
5. copy_to_user: page → program buffer, advance offset, return 4096

Five crossings of boundaries (user/kernel, cache/device, interrupt/thread), most of them invisible to the program, which slept through steps 2–4 and woke with bytes ready. Sockets differ in details (protocol stack instead of filesystem, network card instead of disk) but keep the shape: validate, buffer, queue, interrupt, wake, copy.

The big picture

Syscalls cost crossings, so libraries batch and vectors widen; blocking I/O sleeps honestly and scales poorly; non-blocking plus multiplexing (poll, epoll, io_uring) scales one thread to thousands of descriptors; buffers at every level trade patience for throughput; and one read descends through cache, driver, DMA, and interrupt before returning with bytes.

Every layer so far assumed a running kernel: processes scheduled, pages translated, files mounted, devices humming. None of it explains how the machine gets there from a dead power-on, with RAM empty and no kernel anywhere. Booting into the first process, and main(), is the capstone’s subject.

Footnotes

1. Blocking (computing) - Wikipedia

2. I/O multiplexing - Wikipedia