bnema.dev

Troubleshooting

Common issues and solutions when using Gordon.

Registry Issues

"unauthorized: authentication required"

Cause: Registry authentication is enabled but credentials are missing or invalid.

Solutions:

  1. Login to registry:

    docker login registry.mydomain.com

  2. Check token is valid:

    gordon auth token list

  3. Generate new token:

    gordon auth token generate --subject myuser --expiry 0

"connection refused" on push

Cause: Gordon is not running or registry port is blocked.

Solutions:

  1. Check Gordon is running:

    systemctl --user status gordon
# or
pgrep -f "gordon serve"

  2. Check registry port is accessible:

    curl -v http://localhost:5000/v2/

  3. Check firewall:

    sudo firewall-cmd --list-ports

"server gave HTTP response to HTTPS client"

Cause: Docker/Podman defaults to HTTPS but your registry is serving plain HTTP (common with localhost:5000 or when TLS is terminated by a reverse proxy).

Solution — Docker:

Add the registry to /etc/docker/daemon.json:

{
  "insecure-registries": ["gordon.mydomain.com:5000"]
}

Then restart Docker:

sudo systemctl restart docker

Solution — Podman:

Create or edit ~/.config/containers/registries.conf.d/gordon.conf:

[[registry]]
location = "gordon.mydomain.com:5000"
insecure = true

No restart needed — Podman reads this on each pull/push.

Note: For localhost:5000, Docker already allows insecure access by default. This is only needed for remote or domain-based registry access over HTTP.

TLS certificate errors with the Gordon CLI

Cause: The Gordon server uses a self-signed certificate or TLS is terminated elsewhere.

Solutions (in priority order):

  1. Per-command flag:

    gordon push myapp:latest --insecure

  2. Environment variable:

    export GORDON_INSECURE=true

  3. Per-remote in ~/.config/gordon/remotes.toml:

    [remotes.my-server]
url = "https://gordon.mydomain.com"
token = "..."
insecure_tls = true

  4. Global default in ~/.config/gordon/gordon.toml:

    [client]
insecure_tls = true

"unknown: image not found"

Cause: Image was pushed but no route configured.

Solution: Add route to config:

[routes]
"app.mydomain.com" = "myapp:latest"

Then reload:

gordon reload

Deployment Issues

Container starts but app not accessible

Causes and solutions:

  1. Wrong port exposed: Check your Dockerfile exposes the correct port:

    EXPOSE 3000

  2. Multiple ports - wrong one selected: Add gordon.proxy.port label:

    LABEL gordon.proxy.port=3000

  3. App not listening on 0.0.0.0: Ensure app binds to 0.0.0.0, not 127.0.0.1:

    app.listen(3000, '0.0.0.0');

Container keeps restarting

Cause: Application crashing on startup.

Solutions:

  1. Check container logs:

    docker logs gordon-app-mydomain-com

  2. Check Gordon logs:

    gordon logs -f

  3. Run container manually to debug:

    docker run -it registry.mydomain.com/myapp:latest sh

Environment variables not loaded

Cause: Env file not found or wrong format.

Solutions:

  1. Check file exists with correct name:

    ls ~/.gordon/env/
# Should show: app_mydomain_com.env (dots → underscores)

  2. Check file permissions:

    chmod 600 ~/.gordon/env/app_mydomain_com.env

  3. Check file format (no spaces around =):

    # Correct
KEY=value

# Wrong
KEY = value

Secrets not resolved

Cause: Secret provider not available or secret not found.

Solutions:

  1. For pass backend:

    # Check pass is available
which pass

# Check secret exists
pass show myapp/db-password

  2. For sops backend:

    # Check sops is available
which sops

# Check file can be decrypted
sops -d secrets.yaml

Network Issues

Containers can't reach each other

Cause: Network isolation enabled but services not in same network.

Solutions:

  1. Check attachments are configured:

    [attachments]
"app.mydomain.com" = ["postgres:latest"]

  2. Check containers are in same network:

    docker network inspect gordon-app-mydomain-com

  3. Use correct hostname (image name before colon):

    // Correct
connect("postgresql://postgres:5432/mydb")

// Wrong
connect("postgresql://localhost:5432/mydb")

DNS resolution failing

Cause: Container can't resolve service names.

Solutions:

  1. Verify network isolation is enabled:

    [network_isolation]
enabled = true

  2. Check both containers in same network:

    docker network inspect gordon-app-mydomain-com

Volume Issues

Data not persisting

Cause: Volume not configured or not preserved.

Solutions:

  1. Add VOLUME to Dockerfile:

    VOLUME ["/data"]

  2. Check volume preservation:

    [volumes]
preserve = true  # default: true

  3. List volumes:

    docker volume ls | grep gordon

Disk full

Cause: Registry or logs consuming too much space.

Solutions:

  1. Check disk usage:

    du -sh ~/.gordon/*

  2. Prune unused images:

    docker image prune -a

  3. Configure log rotation:

    [logging.file]
max_size = 100
max_backups = 3

Configuration Issues

Config not reloading

Cause: Gordon not watching config file.

Solution: Manual reload:

gordon reload

Stale targets.toml in config directory

Cause: Gordon renamed targets.toml to remotes.toml in v2.9. The old file is ignored but may cause confusion.

Solution: Delete the old file and use remotes.toml:

rm ~/.config/gordon/targets.toml

If you had remotes configured in targets.toml, recreate them:

gordon remotes add my-server https://gordon.mydomain.com --token YOUR_TOKEN

"failed to read config file"

Cause: TOML syntax error.

Solution: Validate TOML:

# Check for syntax errors
cat ~/.config/gordon/gordon.toml | tomlv
# or use online validator

Logging Issues

No logs appearing

Cause: File logging not enabled.

Solution: Enable in config:

[logging.file]
enabled = true
path = "~/.gordon/logs/gordon.log"

Container logs missing

Cause: Container log collection disabled.

Solution:

[logging.container_logs]
enabled = true  # default: true

Diagnostic Commands

# Check Gordon status
systemctl --user status gordon

# View Gordon logs
gordon logs -f
journalctl --user -u gordon -f

# List containers
docker ps -f "label=gordon.managed=true"

# List networks
docker network ls | grep gordon

# List volumes
docker volume ls | grep gordon

# Check container logs
docker logs gordon-app-mydomain-com

# Inspect container
docker inspect gordon-app-mydomain-com

# Check connectivity
curl -v http://localhost:5000/v2/
curl -v http://localhost:8080/

Getting Help