Linux userspace basics

Quick Cap

Linux userspace is everything above the kernel — the root filesystem, init system, shell utilities, libraries, and application environment. On an embedded device, this is stripped to the minimum: BusyBox replaces hundreds of GNU utilities with a single multi-call binary, a lightweight init system manages services, and the rootfs is often a read-only compressed image (squashfs) with a writable overlay. Understanding this environment is what separates an "embedded Linux developer" from someone who just knows desktop Linux.

Key Facts:

• BusyBox provides 300+ utilities in a single ~1 MB binary via symlink farm (/bin/ls → /bin/busybox)
• Root filesystem for embedded can be as small as 4-8 MB (BusyBox + musl libc), compared to 2+ GB for desktop
• Filesystem types: squashfs (read-only, compressed), UBIFS (NAND flash), ext4 (eMMC/SD), tmpfs (RAM)
• Init systems: BusyBox init (tiny), SysVinit (legacy), systemd (full-featured but 10+ MB overhead)
• Cross-compilation is mandatory — you build on x86 host for ARM/MIPS/RISC-V target
• Debugging: strace is the single most useful tool for diagnosing embedded Linux application issues

Deep Dive

At a Glance

The Embedded Linux Stack

┌─────────────────────────────────┐
│       Your Application          │
├─────────────────────────────────┤
│   Libraries (musl/glibc, etc.)  │
├─────────────────────────────────┤
│   System Services (init, udev)  │
├─────────────────────────────────┤
│   Root Filesystem (rootfs)      │
│   /bin  /etc  /dev  /proc  /sys │
╞═════════════════════════════════╡  ← syscall boundary
│       Linux Kernel              │
├─────────────────────────────────┤
│    Bootloader (U-Boot)          │
├─────────────────────────────────┤
│       Hardware                  │
└─────────────────────────────────┘

Everything above the syscall boundary is userspace — your code runs here with no direct hardware access. All hardware interaction goes through the kernel via system calls (open, read, write, ioctl, mmap).

Root Filesystem Layout

The root filesystem (rootfs) is the directory tree the kernel mounts at /. On embedded systems, it is typically built by a build system (Yocto, Buildroot) rather than installed from packages.

BusyBox symlink farm: Instead of separate binaries for each utility, BusyBox installs as a single executable. Hundreds of symlinks point to it:

/bin/ls → /bin/busybox
/bin/cp → /bin/busybox
/bin/sh → /bin/busybox
/sbin/init → /bin/busybox

BusyBox checks argv[0] to decide which utility to run. This saves megabytes of Flash compared to individual GNU binaries.

Filesystem Types for Embedded

Choosing the right filesystem is critical — using ext4 on raw NAND Flash will destroy it. Each storage medium has a matching filesystem:

The overlayfs pattern is extremely common in embedded: mount a read-only squashfs as the lower layer and a writable partition (or tmpfs) as the upper layer. Changes appear writable to applications, but the base image is never modified. This enables reliable factory reset (just wipe the upper layer) and safe OTA updates (replace the squashfs image).

Init Systems Compared

The init system is PID 1 — the first userspace process the kernel starts. It manages all other services.

BusyBox init is the simplest: it reads /etc/inittab, runs an rcS startup script, and respawns processes marked ::respawn:. No dependency resolution, no socket activation — just raw simplicity. Perfect for a sensor node with 16 MB RAM.

systemd is the full-featured choice: parallel service startup, automatic restart on failure, resource limits via cgroups, journal logging, socket activation, timer units. But it requires glibc (not musl), adds 10+ MB to the rootfs, and uses more RAM. Used in automotive (AGL), industrial gateways, and any device with enough resources.

The Linux Process Model

Every embedded Linux developer needs to understand how processes work:

fork/exec model: fork() creates a copy of the current process. exec() replaces the copy with a new program. This two-step process is how every program is launched in Linux. The shell does fork(), then the child calls exec("./myapp").

Key signals for embedded:

Daemon pattern: Traditional daemons double-fork to detach from the terminal:

1. fork() — parent exits, child continues
2. setsid() — create new session (detach from controlling terminal)
3. fork() again — ensure process is not a session leader
4. Close stdin/stdout/stderr, redirect to /dev/null or log file
5. chdir("/") — avoid holding a mountpoint busy

Modern systemd services skip this — systemd manages the lifecycle, so Type=simple (just run the program, no forking) is preferred.

Cross-Compilation Essentials

You cannot compile ARM code on an x86 host without a cross-toolchain. The toolchain includes:

The naming convention arm-linux-gnueabihf-gcc encodes: architecture (arm), OS (linux), ABI (gnueabihf = hard-float). Using a mismatched toolchain (e.g., soft-float toolchain for hard-float target) produces binaries that crash with "Illegal instruction."

Debugging Embedded Userspace

strace is the #1 tool: if an application works on the host but fails on the target, strace ./myapp immediately shows which file is missing, which permission is denied, or which ioctl returns an error.

Debugging Story: Application Crashes on Target but Works on Host

A team developed a sensor data logger on their x86 development machine. It compiled, linked, and ran perfectly. When they cross-compiled for the ARM target and deployed it, the application crashed immediately with "No such file or directory" — but the binary was clearly there.

Running strace ./logger on the target revealed the real error: the dynamic linker (/lib/ld-linux-armhf.so.3) was trying to load libpthread.so.0, which existed on the target but was built against a different glibc version than the toolchain. The linker aborted because of a symbol version mismatch.

The fix: rebuild the rootfs with the same glibc version as the cross-toolchain, or use static linking for the application (-static flag). The team switched to musl libc for both the toolchain and rootfs, eliminating the version mismatch class of bugs entirely.

The lesson: "It works on my machine" in embedded Linux usually means a toolchain/rootfs mismatch. Always ensure the toolchain's sysroot matches the target's libraries exactly. Build systems like Yocto and Buildroot handle this automatically — manual toolchain setup is error-prone.

What Interviewers Want to Hear

• You understand the difference between embedded and desktop Linux rootfs (BusyBox, musl, minimal)
• You can choose the right filesystem for the storage medium (squashfs for Flash, UBIFS for NAND, tmpfs for RAM)
• You know when to use BusyBox init vs systemd and can justify the tradeoff
• You understand cross-compilation and toolchain basics
• You can debug userspace issues with strace and /proc
• You know the overlayfs pattern for read-only rootfs with writable overlay

Interview Focus

Classic Interview Questions

Q1: "How does an embedded Linux rootfs differ from a desktop Linux installation?"

Model Answer Starter: "An embedded rootfs is purpose-built and minimal — typically 4-50 MB vs 2+ GB for desktop. It uses BusyBox (one binary providing 300+ utilities via symlinks) instead of GNU coreutils, musl libc instead of glibc to save space, and often has no package manager — updates are done by replacing the entire rootfs image. The filesystem is typically squashfs (read-only, compressed) with an overlayfs writable layer, rather than ext4 on a large disk. There's usually no graphical desktop, no multi-user login — just the application and the minimum services to support it."

Q2: "Which filesystem would you choose for a product that uses NAND Flash? Why not ext4?"

Model Answer Starter: "UBIFS on top of UBI (Unsorted Block Images). NAND Flash has unique constraints: pages must be erased before writing, erase blocks wear out after 10K-100K cycles, and bad blocks can appear at any time. UBI handles wear leveling and bad block management. UBIFS sits on top of UBI and provides a POSIX-compatible filesystem. ext4 assumes the storage controller handles these concerns (true for eMMC which has a built-in Flash Translation Layer, but raw NAND has no FTL). Using ext4 on raw NAND would cause uneven wear, no bad block handling, and eventual data corruption."

Q3: "Compare BusyBox init and systemd — when would you use each?"

Model Answer Starter: "BusyBox init is about 10 KB and reads a simple /etc/inittab file. It starts services sequentially via shell scripts and can respawn crashed processes. It has no dependency resolution, no logging infrastructure, no cgroups. I use it for resource-constrained devices with under 32 MB RAM where boot time and memory footprint matter more than service management features. systemd is 10+ MB and requires glibc, but provides parallel startup, automatic dependency resolution, socket activation, journal logging, and cgroup-based resource limits. I use it when the device has 64+ MB RAM and needs complex service management — automotive, industrial gateways, smart home hubs."

Q4: "An application works on your development host but crashes on the ARM target. How do you debug it?"

Model Answer Starter: "First, I run strace ./myapp on the target to see which system call fails. The most common causes are: missing shared library (strace shows ENOENT on a .so file), toolchain/rootfs version mismatch (symbol version error), missing device node in /dev, or incorrect file permissions. If strace shows the app reaching main() but crashing later, I use gdbserver on the target connected to gdb on the host for step-through debugging. I also check dmesg for kernel-level errors like OOM kills or segfault addresses."

Q5: "What is overlayfs and why is it useful in embedded Linux?"

Model Answer Starter: "Overlayfs layers a writable filesystem on top of a read-only one. The lower layer is typically a squashfs image containing the base rootfs — compressed, immutable, and verified. The upper layer is a writable partition (ext4 or tmpfs) that captures all modifications. Applications see a normal read-write filesystem, but the base image is never touched. This enables three things: reliable factory reset (wipe the upper layer), safe OTA updates (replace the squashfs image while the system runs from the old one), and filesystem integrity (the read-only base cannot be corrupted by crashes or power loss)."

Trap Alerts

• Don't say: "I just use Ubuntu/Debian on the target" — that is not embedded Linux, it is desktop Linux on an ARM board
• Don't forget: Cross-compilation — you cannot run apt install on most embedded targets; the rootfs is built on the host
• Don't ignore: Filesystem selection based on storage type — ext4 on raw NAND is a critical mistake

Follow-up Questions

• "How would you minimize the rootfs size for a device with only 8 MB Flash?"
• "What is the difference between glibc, musl, and uClibc-ng? When does the choice matter?"
• "How do you handle persistent configuration on a read-only rootfs?"
• "What is mdev vs udev? When would you use each?"
• "How do you implement over-the-air (OTA) updates for an embedded Linux device?"

Practice

Real-World Tie-In

IoT Sensor Gateway — A LoRaWAN gateway runs on a 64 MB RAM ARM9 board with 128 MB NAND Flash. The rootfs is a 12 MB squashfs image (BusyBox + musl + custom gateway app) with a 16 MB UBIFS writable partition for configuration and logs. BusyBox init starts the gateway service and a watchdog feeder. OTA updates replace the squashfs image via a dual-bank scheme. Total boot time: 3.5 seconds from power-on to first LoRa packet received.

Automotive Telematics Unit — A vehicle telematics box runs on a Cortex-A7 with 512 MB RAM and 4 GB eMMC. Uses systemd for managing 12 services (CAN interface, GPS, cellular modem, OBD-II reader, cloud sync). The rootfs is ext4 with dm-verity for integrity verification. A Yocto-built SDK lets the team cross-compile and deploy new versions. strace and remote gdb are the primary debugging tools during field trials.