In operating systems, a "service" refers to a background resident process or daemon — programs that automatically start when the system boots, work silently in the background, and provide various functions (such as time synchronization, firewalls, scheduled tasks, web servers, databases, etc.). Modern Linux uses systemd to uniformly manage these services, providing powerful dependency management, parallel startup, log integration, and more. This post starts from systemd's core concepts, dives deep into systemctl command usage, explains configuration and troubleshooting for common services (time synchronization, firewall, cron, SSH), and teaches you how to create custom services and make your own programs start automatically on boot. If you're a sysadmin or need to manage Linux servers, this article will upgrade you from "can start/stop services" to "can write custom services, can troubleshoot, can optimize startup order."

systemd Core Concepts: Why Use It, What Problems Does It Solve

Why We Need a Service Manager

In operating systems, a "service" refers to those programs that run in the background for extended periods, providing certain functions to the system or users. Examples include:

Time synchronization service (ntpd / ntpsec): Periodically syncs with time servers to ensure system clock accuracy
Firewall service (firewalld / iptables): Controls network traffic, protects system security
SSH service (sshd): Allows remote login
Web server (nginx / apache): Handles HTTP requests, provides website access
Database service (mysql / postgresql): Stores and queries data
Cron service (crond): Automatically executes scripts according to schedule

These services typically need: 1. Auto-start on boot (no manual start needed) 2. Auto-restart after crash (improves availability) 3. Dependency management (e.g., web service depends on network service, network service depends on network card driver) 4. Log integration (easier troubleshooting) 5. Resource limits (prevent a service from consuming too much CPU/memory)

A service manager is the tool used to uniformly manage these services.

systemd vs SysV init (Legacy Systems)

Before systemd appeared, Linux used SysV init as the service manager (CentOS 6, Ubuntu 14.04 and earlier). Its characteristics:

Service scripts located in /etc/init.d/ directory
Used service <service> start/stop/restart to manage services
Used chkconfig to configure auto-start on boot
Serial startup: Services start one by one in order, slow (especially when there are many services)
Dependency relationships handled manually in scripts: Error-prone

systemd is the service manager for modern Linux distributions (CentOS 7+, Ubuntu 16.04+, Debian 8+), with advantages:

Parallel startup: Multiple services start simultaneously, much faster
Automatic dependency handling: systemd knows service dependencies, automatically starts in order
Unified logging system (journald): All service logs centrally managed, easy to query
Socket activation: Some services can start on-demand (only start when there's a connection, saves resources)
Cgroup integration: Can limit service CPU/memory usage
Unified commands: systemctl one command manages all services

Compatibility: systemd is backward compatible with SysV init commands (service and chkconfig still work on systemd systems, but internally call systemctl).

systemd Core Concept: unit

In systemd, everything is a unit. A unit is systemd's smallest management unit. Common unit types:

Unit Type	Description	Example
service	Most common type, represents a background service (process)	`sshd.service`, `nginx.service`
socket	Represents an IPC or network socket for on-demand service activation	`sshd.socket`, `docker.socket`
target	Represents a group of units (similar to SysV runlevel)	`multi-user.target`, `graphical.target`
mount	Represents a mount point (filesystem)	`home.mount`, `tmp.mount`
timer	Represents a scheduled task (replaces cron)	`logrotate.timer`, `apt-daily.timer`
device	Represents a device (like USB, network card)	`dev-sda.device`
path	Monitors file/directory changes, triggers service startup	`systemd-tmpfiles-clean.path`

Most commonly used is the service type (.service suffix), so when we talk about "services," we usually mean service units.

systemd Startup Process Overview

Kernel boots: After kernel loads, starts systemd (PID 1, first user-space process)
systemd reads config: Reads unit files from /etc/systemd/system/ and /usr/lib/systemd/system/
Determines startup target: Usually multi-user.target (multi-user text mode) or graphical.target (graphical interface)
Parses dependencies: Based on Requires, After, Before fields in unit files, determines startup order
Starts services in parallel: Simultaneously starts multiple services with no dependencies (improves startup speed)
Enters target state: After all dependent services start, system enters target state (like login prompt)

systemctl Commands: The "Muscle Memory" of Daily Use

systemctl is systemd's main command-line tool for managing services. This section covers the commands you'll use most frequently in daily operations.

Basic Operations (Start, Stop, Restart, Check Status)

# Start service (takes effect immediately, but doesn't persist after reboot)
systemctl start <service>

# Stop service
systemctl stop <service>

# Restart service (stops then starts)
systemctl restart <service>

# Reload config files (doesn't stop service, suitable for services that support hot reload like nginx)
systemctl reload <service>

# Check service status (whether running, PID, recent logs, auto-start on boot)
systemctl status <service>

Examples:

1
2
3

systemctl start sshd       # Start SSH service
systemctl status sshd      # Check SSH service status
systemctl reload nginx     # Reload nginx config (without stopping service)

Auto-Start on Boot Management

# Enable auto-start on boot (creates symlink to /etc/systemd/system/multi-user.target.wants/)
systemctl enable <service>

# Disable auto-start on boot (removes symlink)
systemctl disable <service>

# Check if service is enabled for auto-start
systemctl is-enabled <service>

# One-step: start service + enable auto-start
systemctl enable --now <service>

# One-step: stop service + disable auto-start
systemctl disable --now <service>

Examples:

1
2
3

systemctl enable sshd      # Enable SSH service auto-start on boot
systemctl is-enabled sshd  # Output: enabled
systemctl disable firewalld  # Disable firewall auto-start on boot

View Service Lists

# List all running services
systemctl list-units --type=service --state=running

# List all services (including not running)
systemctl list-units --type=service --all

# List all failed services
systemctl list-units --type=service --state=failed

# Search for specific service (like services containing ssh)
systemctl list-units --type=service --all | grep ssh

Example output:

1 2	● sshd.service loaded active running OpenBSD Secure Shell server ● nginx.service loaded active running A high performance web server

View Service Details

# View service unit file content
systemctl cat <service>

# View service dependencies
systemctl list-dependencies <service>

# View service startup time (performance analysis)
systemd-analyze blame
systemd-analyze critical-chain <service>

Examples:

1
2
3

systemctl cat sshd.service  # View sshd unit file
systemctl list-dependencies sshd.service  # View what services sshd depends on
systemd-analyze blame  # View all services' startup time (sorted by duration)

Custom Services: Making Your Programs Auto-Start on Boot

Suppose you have a Python script or compiled binary program that you want to auto-start on boot and auto-restart after crashes. How? Write a systemd unit file.

Minimal Working Example

Suppose you have a script /usr/local/bin/myapp.sh:

#!/bin/bash
while true; do
    echo "MyApp is running..."
    sleep 10
done

Create unit file /etc/systemd/system/myapp.service:

[Unit]
Description=My Custom Application
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/myapp.sh
Restart=always
RestartSec=10
User=nobody
Group=nogroup

[Install]
WantedBy=multi-user.target

Then start and enable:

sudo systemctl daemon-reload  # Reload systemd config
sudo systemctl start myapp
sudo systemctl enable myapp
sudo systemctl status myapp

Unit File Explained

[Unit] Section (Description and Dependencies)

[Unit]
Description=My Custom Application  # Description (shown in systemctl status)
After=network.target               # Start after network service
Before=another.service             # Start before another service
Requires=some.service              # Strong dependency (if dependency fails, this service fails)
Wants=some.service                 # Weak dependency (if dependency fails, this service still tries to start)

[Service] Section (Service Startup Method and Behavior)

[Service]
Type=simple                        # Most common type (considered ready after start)
# Type=forking                     # Suitable for services that fork to background (like traditional daemons)
# Type=oneshot                     # Suitable for one-time execution tasks (like init scripts)

ExecStart=/usr/local/bin/myapp.sh  # Start command (must be absolute path)
ExecReload=/bin/kill -HUP $MAINPID # Reload config (optional)
ExecStop=/bin/kill -TERM$MAINPID  # Stop command (optional, default sends SIGTERM)

Restart=always                     # Auto-restart after crash (always / on-failure / no)
RestartSec=10                      # Wait 10 seconds before restart

User=nobody                        # Which user to run as (security, don't use root)
Group=nogroup                      # Which group to run as

# Resource limits
LimitNOFILE=65536                  # Max open files
LimitNPROC=4096                    # Max processes
MemoryLimit=512M                   # Memory limit (requires cgroup v2 support)
CPUQuota=50%                       # CPU limit (50% means max 0.5 core)

# Working directory
WorkingDirectory=/var/lib/myapp    # Working directory at startup

# Environment variables
Environment="VAR1=value1"
Environment="VAR2=value2"
EnvironmentFile=/etc/myapp/env     # Load environment variables from file

[Install] Section (Auto-Start Configuration)

1
2
3

[Install]
WantedBy=multi-user.target         # Start in multi-user mode (text mode)
# WantedBy=graphical.target        # Start in graphical mode

Common Type Differences

Type	Description	Use Case
simple	Considered ready immediately after start (default)	Most services (like nginx, python scripts)
forking	Service forks a child process then exits (parent exits, child becomes main process)	Traditional daemon programs (like old Apache)
oneshot	Exits after execution (not a resident process)	One-time init tasks (like mounting filesystems)
notify	Service actively notifies systemd "I'm ready" after start (requires program support for sd_notify)	Advanced services (like systemd-networkd)

Must Reload After Modifying Unit File

1 2	sudo systemctl daemon-reload # Tell systemd to re-read unit files sudo systemctl restart myapp

journalctl: systemd's Unified Logging System

systemd comes with journald; all service logs are centrally stored in /var/log/journal/ or /run/log/journal/ (cleared after reboot). Query with journalctl.

Basic Usage

# View all logs (earliest to newest)
journalctl

# View logs for a specific service
journalctl -u <service>

# Real-time view logs (like tail -f)
journalctl -u <service> -f

# View last N lines of logs
journalctl -u <service> -n 50

# View logs from last 1 hour
journalctl -u <service> --since "1 hour ago"

# View logs for a time period
journalctl -u <service> --since "2025-02-01 00:00:00" --until "2025-02-01 23:59:59"

# View boot logs
journalctl -b  # -b 0 means current boot, -b -1 means previous boot

# View kernel logs
journalctl -k

Advanced Usage

# View logs with detailed info (including PID, UID, etc.)
journalctl -u <service> -o verbose

# View logs showing only errors and more severe messages
journalctl -u <service> -p err

# View all logs for a specific user
journalctl _UID=1000

# View logs for a specific executable
journalctl /usr/bin/python3

# Clean logs (keep last 7 days)
journalctl --vacuum-time=7d

# Clean logs (keep last 1GB)
journalctl --vacuum-size=1G

Log Persistence

By default, journald logs are stored in /run/log/journal/ (in memory, cleared after reboot). To persist logs:

Create directory:

1 2	sudo mkdir -p /var/log/journal sudo systemd-tmpfiles --create --prefix /var/log/journal

Restart journald:

1	sudo systemctl restart systemd-journald

Afterward, logs will be saved in /var/log/journal/, persisting after reboot.

Common Service Configuration and Troubleshooting

This section explains configuration and troubleshooting methods for several of the most common system services.

Time Synchronization Service (ntpd / ntpsec / timedatectl)

Why Time Synchronization Is Needed

In distributed systems, clusters, log analysis, and other application scenarios, time synchronization is critical:

Scheduled task execution (like cron jobs): If time is inconsistent, tasks may execute early or late
Log comparison and analysis: If multiple servers have inconsistent time, log timestamps will be chaotic, impossible to correlate
Data synchronization (like database master-slave replication): Inconsistent time can cause data order confusion
Security protocols (like Kerberos authentication, TLS certificates): Too large time 差 causes authentication failure

Solution 1: Using ntpsec (Recommended)

ntpsec is the modernized version of traditional ntpd, with higher security and cleaner code.

Install and start:

1
2
3

sudo apt install ntpsec  # Debian/Ubuntu
sudo yum install ntpsec  # CentOS/RHEL
sudo systemctl enable --now ntpsec

Configuration: Edit /etc/ntpsec/ntp.conf, add time servers (recommend using domestic servers for lower latency):

1
2
3

server ntp.aliyun.com iburst prefer
server ntp.tencent.com iburst
server cn.pool.ntp.org iburst

iburst: Fast sync at startup (sends 8 packets instead of 1)
prefer: Prioritize this server

Restart service:

1	sudo systemctl restart ntpsec

Check sync status:

1	ntpq -p # View currently connected time servers

Solution 2: Using timedatectl (Simpler)

timedatectl is systemd's built-in time management tool, using systemd-timesyncd for NTP synchronization underneath.

Enable NTP sync:

1	sudo timedatectl set-ntp true

Check status:

1	timedatectl

Example output:

               Local time: Mon 2025-02-03 12:00:00 UTC
           Universal time: Mon 2025-02-03 12:00:00 UTC
                 RTC time: Mon 2025-02-03 12:00:00
                Time zone: UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Set timezone (if needed):

1	sudo timedatectl set-timezone Asia/Shanghai

View available timezones:

1	timedatectl list-timezones \| grep Shanghai

ntpdate (One-Time Sync, Not Recommended for Resident Use)

If you only need to sync time once temporarily (not a resident service):

1	sudo ntpdate ntp.aliyun.com

Disadvantages:

Brutally modifies time (direct jump), may cause errors in time-dependent programs (like cron, database sync)
ntpd / ntpsec uses smooth adjustment, avoids time jumps

Recommendation: Use ntpsec or timedatectl for long-running synchronization, avoid ntpdate's issues.

Firewall Service (firewalld)

Basic Introduction

Firewall is a network security tool used to control and filter network traffic, preventing unauthorized access. Common firewalls in Linux:

iptables (traditional, rules are ordered, complex configuration)
firewalld (modern, dynamic rules, supports zone concept, RHEL/CentOS default)
ufw (Ubuntu-friendly firewall, simplified iptables)

firewalld is a dynamic firewall based on iptables/nftables, providing a friendlier command-line interface.

Enable and Disable

sudo systemctl start firewalld
sudo systemctl enable firewalld
sudo systemctl status firewalld

sudo systemctl stop firewalld
sudo systemctl disable firewalld

Basic Rule Operations

Open/Close Ports

# Open port 80 (TCP)
sudo firewall-cmd --permanent --add-port=80/tcp
sudo firewall-cmd --reload

# Open port range 80-90
sudo firewall-cmd --permanent --add-port=80-90/tcp
sudo firewall-cmd --reload

# Close port 80
sudo firewall-cmd --permanent --remove-port=80/tcp
sudo firewall-cmd --reload

--permanent: Write permanent rule (otherwise expires after reboot)
--reload: Reload rules (make permanent rules take effect)

Open/Close Services

# Open HTTP service (includes port 80)
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --reload

# View currently open services and ports
sudo firewall-cmd --list-all

Common service names: http, https, ssh, ntp, mysql, postgresql, etc.

zone Concept

firewalld uses zones to define different security levels:

Zone Name	Trust Level	Default Policy	Typical Use Case
drop	Lowest	Drop all incoming traffic, no response	Highest security, completely hide device
public	Low	Default deny all inbound, only allow SSH	Public networks (like WiFi hotspots)
work	High	Allow trusted internal network access, default allow SSH, Samba	Company internal network
home	High	Allow home network access	Home network
trusted	Highest	Allow all traffic	Completely trusted environment

View current default zone:

1	sudo firewall-cmd --get-default-zone

Change default zone:

1	sudo firewall-cmd --set-default-zone=work

Scheduled Tasks (crontab)

Scheduled tasks allow the system to automatically execute certain scripts or commands at specified times, for example:

Scheduled database backup (every day at 2 AM)
Scheduled log file or temp folder cleanup (every Sunday)
Scheduled time sync (every hour)

Edit and View

1
2
3

crontab -e  # Edit current user's scheduled tasks
crontab -l  # View current user's scheduled tasks
crontab -r  # Delete all of current user's scheduled tasks

Format

1	MIN HOUR DAY MONTH WEEKDAY command

MIN: Minutes (0-59)
HOUR: Hours (0-23)
DAY: Day of month (1-31)
MONTH: Month (1-12)
WEEKDAY: Day of week (0-7, 0 and 7 both represent Sunday)
command: Command to execute (recommend using absolute paths)

Special symbols:

*: Represents "every" (like * in minutes means every minute)
-: Represents range (like 1-5 means 1 to 5)
/: Represents interval (like */10 means every 10 units)
,: Represents multiple values (like 1,2,6 means 1, 2, 6)

Examples

# Run backup script daily at 1:30 AM
30 1 * * * /usr/local/bin/backup.sh

# Sync time daily at 2:30 AM (redirect output to null, no output)
30 2 * * * /usr/sbin/ntpdate -u ntp.aliyun.com &> /dev/null

# Execute at 5 minutes past every hour
5 * * * * /usr/local/bin/hourly-task.sh

# Execute every 10 minutes
*/10 * * * * /usr/local/bin/monitor.sh

# Execute every Sunday at 3 AM
0 3 * * 0 /usr/local/bin/weekly-cleanup.sh

View Logs

1 2	grep CRON /var/log/syslog # Debian/Ubuntu grep CRON /var/log/cron # CentOS/RHEL

SSH Service Configuration and Security Hardening

SSH is the standard way to remotely manage Linux servers. In "Linux Basics" we already covered basic usage (ssh user@host, password-free login); here we cover some sshd configuration and security hardening.

Configuration File

Configuration file: /etc/ssh/sshd_config

After modification, restart service:

1	sudo systemctl restart sshd

Common Configuration Items

# Listening port (default 22, recommend changing to another port to reduce scanning)
Port 22222

# Whether to allow root user direct login (recommend disabling)
PermitRootLogin no

# Whether to allow password login (recommend disabling, only use key login)
PasswordAuthentication no

# Whether to allow empty password login (strongly recommend disabling)
PermitEmptyPasswords no

# Allow public key authentication
PubkeyAuthentication yes

# Public key file location
AuthorizedKeysFile .ssh/authorized_keys

# Whether to allow X11 forwarding (graphical interface forwarding)
X11Forwarding yes

# Max authentication attempts (prevent brute force)
MaxAuthTries 3

# Idle timeout (seconds)
ClientAliveInterval 300
ClientAliveCountMax 0

Security Hardening Recommendations

Change port: Change default port 22 to another (like 22222), reduce scan probability
Disable root login: PermitRootLogin no, only allow regular user login, use sudo when permissions needed
Disable password login: PasswordAuthentication no, only allow key login (prevent brute force)
Install fail2ban: Automatically ban brute force IPs (auto-ban after consecutive failures)
Configure firewall: Only allow specific IPs to connect to SSH (if you have fixed IPs)
Use SSH keys: 4096-bit RSA or Ed25519 keys (much more secure than passwords)

Service Troubleshooting Process: What to Do When Service Won't Start

When a service won't start, follow this troubleshooting process:

1. Check Service Status

1	sudo systemctl status <service>

Focus on:

Active: Whether it shows active (running) or failed
Main PID: Main process PID (if 0, process has exited)
Recent logs: Usually shows last few lines of logs, check for error messages

2. View Detailed Logs

1	sudo journalctl -u <service> -xe

-x: Show extra explanatory information
-e: Jump to end of log (newest logs)

3. Check Configuration File Syntax

Many services provide config file syntax check tools:

nginx: nginx -t
apache: apachectl configtest
sshd: sshd -t

4. Check Port Conflicts

1 2	sudo ss -lntp \| grep <port> # Check if port is occupied sudo lsof -i :<port> # See which process is using the port

5. Check File Permissions

Service may fail to start due to insufficient permissions:

1 2	ls -l /path/to/config ls -l /var/log/<service>

Ensure the user the service runs as (like www-data, nginx, nobody) has permissions to read/write related files.

6. Check SELinux / AppArmor

If SELinux (RHEL/CentOS) or AppArmor (Ubuntu/Debian) is enabled, it may prevent service startup.

Temporarily disable SELinux (only for troubleshooting):

1	sudo setenforce 0 # Set to Permissive mode

View SELinux logs:

1	sudo ausearch -m avc -ts recent

Temporarily disable AppArmor:

1	sudo systemctl stop apparmor

7. Check Dependencies

1	sudo systemctl list-dependencies <service>

If dependent services haven't started, current service also can't start.

Summary and Further Reading

This article covers the core content of systemd service management: 1. ✅ systemd core concepts (unit, dependencies, startup process) 2. ✅ systemctl daily usage (start/stop, auto-start on boot, check status) 3. ✅ Custom services (how to make your program auto-start on boot) 4. ✅ journalctl log management (query, filter, cleanup) 5. ✅ Common service configuration (time sync, firewall, cron, SSH) 6. ✅ Service troubleshooting process (what to do when service won't start)

Further Reading:

systemd official documentation: https://www.freedesktop.org/wiki/Software/systemd/
systemd for Administrators series: http://0pointer.de/blog/projects/systemd-for-admins-1.html
journalctl manual: man journalctl
systemd.service manual: man systemd.service

Next Steps:

《 Linux Package Management 》: Learn how to install/uninstall/update packages (apt/yum/dnf/rpm)
《 Linux Process and Resource Management 》: Learn how to monitor and limit process CPU/memory usage
《 Linux User Management 》: Learn how to manage users/groups/permissions

By this point, you should have upgraded from "can start/stop services" to "can write custom services, can troubleshoot, can optimize startup order." Service management is a core Linux sysadmin skill; mastering systemd allows you to better manage servers.