DevOps · Guide

Linux Basics

Filesystem, permissions, processes, systemd, networking.

— min read DevOps

Theory

Linux Basics: Filesystem, permissions, processes, systemd, networking.

Linux follows the Unix philosophy: everything is a file (devices, sockets, processes in /proc), tools do one thing well, and programs communicate via text streams through pipes. The kernel manages hardware; the shell (bash/zsh/fish) is just a program that interprets commands.

The filesystem hierarchy standard (FHS): /bin and /usr/bin hold executables, /etc holds config files (plain text, editable), /var holds variable data (logs in /var/log, databases in /var/lib), /tmp is cleared on reboot, /proc and /sys are virtual filesystems exposing kernel state.

File permissions use a 3×3 bit matrix: owner/group/other each have read(4)/write(2)/execute(1) bits. chmod 755 file sets rwxr-xr-x. chown user:group transfers ownership. Directories need execute bit to be entered (cd). setuid bit (chmod u+s) runs executable as owner — used by sudo, passwd.

Process management: ps aux lists all processes with PID, CPU, memory. kill -9 PID force-kills; kill -15 (SIGTERM) requests graceful shutdown. top/htop show real-time CPU and memory. systemctl start/stop/restart/status controls systemd units; journalctl -u service -f follows logs.

Networking: ss -tlnp shows listening sockets with PIDs (replaces netstat). ip addr shows interfaces; ip route shows routing table. curl -v tests HTTP endpoints. tcpdump -i eth0 port 80 captures packets. /etc/hosts for local DNS overrides; /etc/resolv.conf for nameservers.

Package management: apt (Debian/Ubuntu) uses apt install/remove/update/upgrade/autoremove. apt update refreshes package index; apt upgrade installs updates. dpkg -l lists installed packages. yum/dnf for RHEL/CentOS/Fedora. snap and flatpak for distribution-agnostic packages.

Cron schedules recurring jobs: crontab -e edits user crontab. Format: minute hour day month weekday command. */5 * * * * runs every 5 minutes. @reboot runs once at startup. Run scripts with absolute paths since cron has a minimal PATH. Redirect stderr: cmd >> /log 2>&1.

SSH: ssh-keygen -t ed25519 generates a key pair. Copy public key with ssh-copy-id user@host. ~/.ssh/config stores per-host settings (IdentityFile, Port, ProxyJump for bastion hops). ssh -L 5432:db:5432 user@bastion creates a local port forward to a database behind a firewall.

Architecture Diagram

Users / clients
         |
  Linux Basics
         |
  Core services
         |
  Data + observability

Examples

bash — file & permissions
# File operations
ls -lah /var/log          # List with human-readable sizes
find / -name "*.conf" 2>/dev/null   # Find all .conf files
grep -rn "error" /var/log/ --include="*.log"  # Recursive search
tail -f /var/log/syslog   # Follow log in real time

# Permissions
chmod 755 script.sh       # rwxr-xr-x (owner full, others read+exec)
chmod 600 ~/.ssh/id_ed25519  # rw------- (private key must be 600)
chown www-data:www-data /var/www  # Transfer ownership to web server user
find /uploads -type f -exec chmod 644 {} \;  # Batch permission fix
bash — process & system
# Process management
ps aux | grep python      # Find Python processes
kill -15 1234             # Graceful shutdown (SIGTERM)
kill -9 1234              # Force kill (SIGKILL) — last resort

# Disk & memory
df -h                     # Disk usage by filesystem
du -sh /var/log/*         # Size of each log directory
free -h                   # Memory usage (used/free/cache/buff)
vmstat 1 5               # CPU/memory stats every 1s, 5 times

# Network
ss -tlnp                  # Listening ports with process names
curl -I https://example.com  # HTTP headers only
lsof -i :8000             # What process owns port 8000?
bash — ssh & systemd
# SSH config (~/.ssh/config)
Host prod-bastion
    HostName 52.1.2.3
    User ubuntu
    IdentityFile ~/.ssh/prod_key

Host db-private
    HostName 10.0.1.50
    User ubuntu
    ProxyJump prod-bastion  # Jump through bastion

# Systemd
systemctl status nginx
systemctl restart nginx
journalctl -u nginx -f        # Follow nginx logs
journalctl --since "1 hour ago" -u nginx  # Time filter
systemctl list-units --failed  # Show failed units

Interview Questions

What does chmod 755 mean in octal notation?

7 (owner) = 4+2+1 = rwx. 5 (group) = 4+0+1 = r-x. 5 (other) = r-x. Common for executables and directories you want others to read/execute but not write. chmod 644 is typical for files (owner can write, others read-only).

How do you find what process is using port 8080?

ss -tlnp | grep :8080 (modern), or lsof -i :8080, or fuser 8080/tcp. ss is preferred over deprecated netstat. Output shows PID in the last column.

What is the difference between kill -9 and kill -15?

SIGTERM (15) asks the process to shut down gracefully — it can catch the signal, flush buffers, and clean up. SIGKILL (9) is sent by the kernel — cannot be caught or ignored, process is immediately terminated. Always try SIGTERM first; use SIGKILL only if the process is unresponsive.

How does sudo work, and what is /etc/sudoers?

sudo runs a command as another user (default: root). /etc/sudoers defines who can run what as whom. Entry format: user hostname=(runas) commands. NOPASSWD: skips password prompt. Use visudo to edit — it validates syntax to prevent lockouts. Sudoers.d/ directory allows per-file additions.

What is the difference between /etc/passwd and /etc/shadow?

/etc/passwd is world-readable and contains user info (username, UID, GID, home, shell) but not passwords (shows x). /etc/shadow is root-only and contains hashed passwords with aging info. This split prevents brute-force attacks from unprivileged users.

How would you debug a service that fails to start?

systemctl status service-name shows last exit code and recent logs. journalctl -u service-name --no-pager -n 50 shows more log context. Check the ExecStart command manually, verify the user the service runs as has permission to its files, and check for port conflicts with ss -tlnp.

What is a zombie process?

A process that has exited but its entry remains in the process table because the parent hasn't called wait() to collect the exit status. Zombies consume a PID but no memory or CPU. Fix: send SIGCHLD to the parent process or kill the parent (zombies are automatically reaped by init/systemd). Large zombie counts indicate a buggy parent.

FAANG: How would you investigate a Linux server that is suddenly slow?

Start with top/htop to identify CPU/memory saturation. Check load average — above core count means CPU-bound. vmstat 1 3 shows context switches and IO wait. iostat -xz 1 3 identifies disk bottlenecks. sar -n DEV 1 3 checks network saturation. netstat -s | grep -i retrans shows TCP retransmits (network issues). strace -p PID shows syscalls for a specific process. The RED method: Rate, Errors, Duration for the system's exposed service.

Best Practices

  • Never run production services as root — create a dedicated service user with minimal permissions (useradd -r -s /bin/false svcname).
  • Use systemctl enable svc so services auto-restart on reboot — don't rely on manual restarts after patching.
  • Rotate logs with logrotate — configure daily rotation with compress and delaycompress to prevent disk-fill incidents.
  • Use SSH key authentication everywhere — disable password auth in sshd_config (PasswordAuthentication no).
  • Lock down sudo — grant only the specific commands each service account needs, not blanket NOPASSWD ALL.
  • Set file permissions on private keys to 600 (chmod 600 ~/.ssh/id_ed25519) — SSH refuses to use world-readable keys.

Common Mistakes

Watch for these patterns — they cause most production incidents around Linux Basics.
  • Setting SSH private key permissions to 644 or 755 — SSH silently refuses to use keys that are group/world-readable, giving a confusing "permission denied" error.
  • Using kill -9 first — always try SIGTERM first to allow clean shutdown. SIGKILL can leave locked files, corrupt data, or zombie processes.
  • Editing /etc/sudoers with a regular editor instead of visudo — a syntax error in sudoers locks all users out of sudo permanently.
  • Running chmod -R 777 /var/www to "fix permissions" — makes every file world-writable, trivially exploitable if any web-accessible script has code execution.
  • Cron jobs with relative paths (script.sh instead of /home/user/script.sh) — cron has a minimal PATH that doesn't include user directories, so jobs silently fail.

Cheat Sheet

ls -lahList with permissions, sizes, hidden files
chmod 755 / 644rwxr-xr-x for dirs, rw-r--r-- for files
ss -tlnpShow listening ports with process names
journalctl -u svc -fFollow logs for a systemd service
systemctl restart svcRestart a service (stop + start)
ps aux | grep nameFind processes by name
kill -15 PIDGraceful shutdown (SIGTERM)
kill -9 PIDForce kill (SIGKILL) — last resort
df -hDisk usage by filesystem (human-readable)
du -sh /path/*Size of each item in a directory
ssh -L 5432:db:5432 user@hostLocal port forward through SSH tunnel
crontab -eEdit current user's cron schedule

Practical Exercises

Service ownership and permissions

Create a user webuser with no login shell. Create /var/www/app/ owned by webuser:webuser with 755 on dirs and 644 on files. Write a systemd unit that runs a Python HTTP server as webuser. Enable and verify it survives a reboot.

SSH tunnel to a locked-down service

Start PostgreSQL on a remote VM with listen_addresses = 'localhost' (no external access). Create an SSH tunnel ssh -L 5432:localhost:5432 user@remote. Connect from your local machine with psql -h localhost -p 5432. Verify you cannot connect without the tunnel.

Incident simulation

Start a web app. Fill disk by writing dd if=/dev/zero of=/tmp/bigfile bs=1M count=4000. Observe what breaks. Identify the failing service via journalctl -u app -f. Free disk by deleting bigfile. Verify recovery. Write a 5-line post-mortem.