Series 7 — Part 2 of 6

Python AI services need to keep running after you close your SSH session. The naive approach fails in three different ways. This article covers the nohup + disown pattern, the stdin redirect requirement, and how to verify a service is actually alive after SSH logout.

Why Simple & Fails

When you start a process with python server.py & and close the SSH session, the process receives SIGHUP (hangup signal) and terminates. This is the shell telling child processes that the controlling terminal is gone.

# This WILL die when SSH session closes
python /home/aiuser/tts/server.py &

# This also WILL die (nohup without stdin redirect)
nohup python /home/aiuser/tts/server.py &

# The correct pattern: nohup + stdin redirect + disown
nohup python /home/aiuser/tts/server.py < /dev/null >> /var/log/tts.log 2>&1 &
disown $!

Why the stdin Redirect Is Required

Without < /dev/null, the process inherits the SSH session's stdin. When the SSH session closes, any attempt by the process to read stdin (even accidentally, via a library) produces an error. The /dev/null redirect gives the process a safe, always-available stdin that returns EOF immediately.

A Complete Startup Script

#!/usr/bin/env bash
# /home/aiuser/start-services.sh

set -euo pipefail

LOG_DIR="/var/log/ai-stack"
mkdir -p "$LOG_DIR"

start_service() {
    local name="$1"
    local cmd="$2"
    local port="$3"

    if ss -tlnp | grep -q ":${port}"; then
        echo "[SKIP] ${name} already running on port ${port}"
        return
    fi

    echo "[START] ${name} on port ${port}"
    nohup $cmd < /dev/null >> "${LOG_DIR}/${name}.log" 2>&1 &
    disown $!

    # Wait for health check
    for i in $(seq 1 30); do
        if curl -sf "http://localhost:${port}/health" > /dev/null 2>&1; then
            echo "[OK] ${name} healthy"
            return
        fi
        sleep 1
    done
    echo "[WARN] ${name} health check timed out"
}

start_service "chromadb"  "chroma run --port 8000"   8000
start_service "kokoro"    "python /home/aiuser/tts/server.py --port 9010"  9010
start_service "whisper"   "python /home/aiuser/stt/server.py --port 9011" 9011
start_service "converter" "python /home/aiuser/audio/server.py --port 9012" 9012

Verifying Services After SSH Logout

# Check if port is listening
ss -tlnp | grep ':9010'

# Check process is running
pgrep -f 'tts/server.py'

# Health check via curl
curl -s http://localhost:9010/health

# Check it survived SSH logout by opening a NEW SSH session and running the above

What to Watch For

  • Log file growth — Services logging verbosely will fill disk. Add logrotate config for each service log file.
  • Startup on reboot — nohup + disown doesn't survive a reboot. Add your startup script to crontab with @reboot /home/aiuser/start-services.sh or create systemd unit files for services that must survive reboots.
  • Zombie processes — If a service crashes and is restarted, the old port binding may linger. The ss -tlnp | grep :port check prevents starting a duplicate.