bnema.dev

Troubleshooting

Common issues and solutions when using Gordon.

Registry Issues

"unauthorized: authentication required"

Cause: Registry authentication is enabled but credentials are missing or invalid.

Solutions:

  1. Login to registry:

    docker login registry.mydomain.com

  2. Check token is valid:

    gordon auth token list

  3. Generate new token:

    gordon auth token generate --subject myuser --expiry 0

"connection refused" on push

Cause: Gordon is not running or registry port is blocked.

Solutions:

  1. Check Gordon is running:

    systemctl --user status gordon
# or
pgrep -f "gordon serve"

  2. Check registry port is accessible:

    curl -v http://localhost:5000/v2/

  3. Check firewall:

    sudo firewall-cmd --list-ports

"server gave HTTP response to HTTPS client"

Cause: Docker/Podman defaults to HTTPS but your registry is serving plain HTTP (common with localhost:5000 or when TLS is terminated by a reverse proxy).

Solution — Docker:

Add the registry to /etc/docker/daemon.json:

{
  "insecure-registries": ["gordon.mydomain.com:5000"]
}

Then restart Docker:

sudo systemctl restart docker

Solution — Podman:

Create or edit ~/.config/containers/registries.conf.d/gordon.conf:

[[registry]]
location = "gordon.mydomain.com:5000"
insecure = true

No restart needed — Podman reads this on each pull/push.

Note: For localhost:5000, Docker already allows insecure access by default. This is only needed for remote or domain-based registry access over HTTP.

TLS certificate errors with the Gordon CLI

Cause: The Gordon server uses a self-signed certificate or TLS is terminated elsewhere.

Solutions (in priority order):

  1. Per-command flag:

    gordon images push myapp:latest --insecure

  2. Environment variable:

    export GORDON_INSECURE=true

  3. Per-remote in ~/.config/gordon/remotes.toml:

    [remotes.my-server]
url = "https://gordon.mydomain.com"
token = "..."
insecure_tls = true

  4. Global default in ~/.config/gordon/gordon.toml:

    [client]
insecure_tls = true

"unknown: image not found"

Cause: The app manifest references an image that is unavailable to the runtime.

Solution: Push the image to the configured registry, verify the service's image reference in the app manifest, then apply and deploy the accepted revision:

gordon apps apply --file app.toml
gordon apps deploy app

Deployment Issues

Container starts but app not accessible

Causes and solutions:

  1. Wrong port exposed: Check your Dockerfile exposes the correct port:

    EXPOSE 3000

  2. Multiple ports - wrong one selected: Add gordon.proxy.port label:

    LABEL gordon.proxy.port=3000

  3. App not listening on 0.0.0.0: Ensure app binds to 0.0.0.0, not 127.0.0.1:

    app.listen(3000, '0.0.0.0');

Container keeps restarting

Cause: Application crashing on startup.

Solutions:

  1. Check workload logs:

    gordon apps logs blog --service web

  2. Check Gordon logs:

    gordon daemon logs -f

  3. Run container manually to debug:

    docker run -it registry.mydomain.com/myapp:latest sh

Environment variables not loaded

Cause: The variable is not declared in the app file, or its secret value was never set.

Solutions:

  1. Declare app-wide public values under [env], per-service public values under [services.<name>.env], and sensitive values under [services.<name>.secrets] in the app file, then apply it:

    gordon apps apply --file ./blog.toml

  2. Check which secret values are set (names only):

    gordon apps secrets list blog

  3. Deploy or restart: values apply on the next deploy/restart, never to running containers.

    gordon apps restart blog

Secrets not resolved

Cause: Secret provider not available or secret not found.

Solutions:

  1. For pass backend:

    # Check pass is available
which pass

# Check secret exists
pass show myapp/db-password

  2. For sops backend:

    # Check sops is available
which sops

# Check file can be decrypted
sops -d secrets.yaml

Network Issues

Containers can't reach each other

Cause: Network isolation enabled but services not in same network.

Solutions:

  1. Check that both services declare the same shared network in the app manifest.

  2. Check containers are attached to that network:

    docker network inspect NETWORK_NAME

  3. Use the service name as the internal hostname:

    // Correct
connect("postgresql://postgres:5432/mydb")

// Wrong
connect("postgresql://localhost:5432/mydb")

DNS resolution failing

Cause: Container can't resolve service names.

Solutions:

  1. Verify network isolation is enabled:

    [network_isolation]
enabled = true

  2. Check both containers in same network:

    docker network inspect gordon-app-mydomain-com

Volume Issues

Data not persisting

Cause: Volume not configured or not preserved.

Solutions:

  1. Add VOLUME to Dockerfile:

    VOLUME ["/data"]

  2. Check volume preservation:

    [volumes]
preserve = true  # default: true

  3. List volumes:

    docker volume ls | grep gordon

Disk full

Cause: Registry or logs consuming too much space.

Solutions:

  1. Check disk usage:

    du -sh ~/.gordon/*

  2. Prune unused images:

    docker image prune -a

  3. Configure log rotation:

    [logging.file]
max_size = 100
max_backups = 3

Configuration Issues

Config not reloading

Cause: Gordon not watching config file.

Solution: Manual reload:

gordon daemon reload

Stale targets.toml in config directory

Cause: Gordon renamed targets.toml to remotes.toml in v2.9. The old file is ignored but may cause confusion.

Solution: Delete the old file and use remotes.toml:

rm ~/.config/gordon/targets.toml

If you had remotes configured in targets.toml, recreate them:

gordon remotes add my-server https://gordon.mydomain.com --token YOUR_TOKEN

"failed to read config file"

Cause: TOML syntax error.

Solution: Validate TOML:

# Check for syntax errors
cat ~/.config/gordon/gordon.toml | tomlv
# or use online validator

Logging Issues

No logs appearing

Cause: File logging not enabled.

Solution: Enable in config:

[logging.file]
enabled = true
path = "~/.gordon/logs/gordon.log"

Workload logs unavailable

gordon apps logs APP --service SERVICE reads directly from the container runtime. Confirm that the app has an active deployment, use the exact service name, and check the runtime's logging driver and retention settings. Gordon does not write workload logs to logging.container_logs files.

Diagnostic Commands

# Check Gordon status
systemctl --user status gordon

# View Gordon logs
gordon daemon logs -f
journalctl --user -u gordon -f

# List containers
docker ps -f "label=gordon.managed=true"

# List networks
docker network ls | grep gordon

# List volumes
docker volume ls | grep gordon

# Check workload logs through Gordon
gordon apps logs blog --service web

# Inspect runtime containers when diagnosing locally
docker ps -f "label=gordon.app=blog"

# Check connectivity
curl -v http://localhost:5000/v2/
curl -v http://localhost:8088/

Getting Help