Telemetry Configuration

Export traces, metrics, and logs from Gordon via OpenTelemetry (OTLP/HTTP).

When enabled, Gordon ships all three observability signals to an OTLP-compatible backend such as OpenObserve, Jaeger, or Grafana Cloud. Existing zerolog output continues unchanged; the OTel log bridge runs alongside it.

Configuration

[telemetry]
enabled = false
endpoint = ""
auth_token = ""
traces = true
metrics = true
logs = true
trace_sample_rate = 1.0

Options

Option Type Default Description
enabled bool false Enable OTLP telemetry export
endpoint string "" OTLP HTTP endpoint URL (e.g. http://localhost:4318/api/default)
auth_token string "" Base64-encoded user:password for Basic auth
traces bool true Export distributed traces
metrics bool true Export metrics (deploy counters, container lifecycle, registry, events)
logs bool true Export Gordon, proxy access, and app container logs to OTLP
trace_sample_rate float 1.0 Fraction of traces to sample (0.0 = none, 1.0 = all)

How It Works

Gordon initializes an OTel provider at startup. When telemetry is enabled:

  1. Traces -- Spans wrap critical operations: container deploy, image pull, registry manifest push, and proxy target resolution. The otelhttp middleware adds a span to every HTTP request on both the proxy and registry servers.
  2. Metrics -- Gordon records custom counters and histograms for deploys, container restarts, crash loops, managed container count, registry pushes, and event bus throughput.
  3. Logs -- Gordon exports three log families (see Logs).

When telemetry is disabled (the default), all OTel instruments are noop -- zero overhead.

Logs

With logs = true, Gordon exports:

Family Source service.name
Gordon process logs zerowrap output, with trace_id/span_id inside traced requests gordon
Proxy access logs Every proxied request, attributed to the app service owning the host <app>.<service> (gordon for registry and unknown hosts)
App container logs stdout and stderr of every running app service container <app>.<service>

Every app record carries these resource attributes:

Attribute Example Use
service.name blog.web One exact service; unique across apps
service.namespace blog All services of one app
gordon.app blog Filter by app
gordon.service web Filter by service name across apps

Record attributes:

  • log.type: access or container
  • log.iostream: stdout or stderr (container logs)
  • Access logs: http.request.method, http.response.status_code, server.address, url.path, client.address, request.id, and more. Severity follows the status: 5xx is ERROR, 4xx is WARN, otherwise INFO.

Loki indexes service.name and service.namespace as stream labels by default; the other attributes are structured metadata, filtered after the stream selector. Example queries in Grafana:

{service_namespace="blog"}                          # every log of the blog app
{service_name="blog.web"} | log_type="access"       # access logs of one service
{service_namespace=~".+"} | gordon_service="web"    # every "web" service, all apps

Container output is exported from the moment Gordon starts, including the first lines of every container deployed afterwards. Output emitted while Gordon is stopped is not exported. Lines longer than 64 KiB are truncated. Export is buffered and never slows the proxy or the app: on overload, the oldest records are dropped.

Per-app opt-out

App container logs are exported by default. Opt out in the app manifest, per app or per service; a service setting overrides the app setting:

[telemetry]
logs = true          # app default

[services.db.telemetry]
logs = false         # only "db" stops exporting

The opt-out covers container output only; access logs of the app's hosts are still exported.

Endpoint URL

The endpoint field accepts a full URL. Gordon parses it to extract:

  • Host and port for the OTLP exporter
  • Base path appended with /v1/traces, /v1/metrics, or /v1/logs
  • Scheme -- http:// disables TLS; https:// enables it

For OpenObserve, the endpoint typically includes the organization path:

http://localhost:5080/api/default

This produces export URLs like http://localhost:5080/api/default/v1/traces.

Authentication

Set auth_token to the Base64-encoded user:password string. Gordon sends it as a Basic authorization header on every OTLP export request.

For OpenObserve, copy the token from Ingestion > OTLP in the web UI.

Gordon itself is the platform, not an app, so the token does not go through gordon apps secrets. Store it with one of these methods:

Method How
Environment variable GORDON_TELEMETRY_AUTH_TOKEN=<token> gordon serve
Plain text in config Set auth_token = "<token>" (restrict file permissions)

Trace Sampling

Value Behavior
1.0 Sample every trace (default)
0.5 Sample 50% of traces
0.0 Drop all traces (tracing pipeline stays initialized)

At low traffic volumes (< 1000 req/s), keep the rate at 1.0. Reduce it if storage costs or export bandwidth become a concern.

Metrics Reference

Gordon exports these custom metrics (all prefixed with gordon.):

Deployments

Metric Type Unit Description
gordon.deploy.total Counter - Total deployments attempted
gordon.deploy.duration_seconds Histogram s Time from deploy start to completion
gordon.deploy.errors Counter - Deployments that failed

Attributes: domain, image

Container Lifecycle

Metric Type Unit Description
gordon.container.restarts Counter - Container restart count
gordon.container.crash_loops Counter - Crash loop detections
gordon.container.managed Gauge - Containers of running app services, read from ACTIVE app state at each export

Attributes: source (restarts only: monitor or api). gordon.container.managed has no attributes; tell instances apart with the host.name resource attribute.

Registry

Metric Type Unit Description
gordon.registry.push.total Counter - Manifest pushes received
gordon.registry.push.bytes Counter By Total manifest bytes pushed

Attributes: name, reference

Event Bus

Metric Type Unit Description
gordon.events.processed Counter - Events handled successfully
gordon.events.dropped Counter - Events dropped (channel full)

Attributes: event_type

HTTP (via otelhttp)

The otelhttp middleware automatically records standard HTTP server metrics on both the proxy and registry servers:

  • http.server.request.duration
  • http.server.request.body.size
  • http.server.response.body.size

Trace Spans

Span Name Package Description
container.deploy container Full deploy lifecycle (root span)
container.create_and_start container Container creation, start, and readiness
container.ensure_image container Image pull and tagging
registry.put_manifest registry Manifest storage and event publish
proxy.get_target proxy Proxy target resolution and caching

The otelhttp middleware adds an HTTP-level span to every request on both servers.

Examples

OpenObserve (self-hosted)

[telemetry]
enabled = true
endpoint = "http://localhost:5080/api/default"
auth_token = "YWRtaW5AZXhhbXBsZS5jb206c2VjcmV0"
traces = true
metrics = true
logs = true
trace_sample_rate = 1.0

Grafana Cloud (OTLP)

[telemetry]
enabled = true
endpoint = "https://otlp-gateway-prod-us-east-0.grafana.net/otlp"
auth_token = "<instance-id>:<api-key>"
traces = true
metrics = true
logs = true
trace_sample_rate = 0.1

Traces Only (minimal overhead)

[telemetry]
enabled = true
endpoint = "http://localhost:4318"
traces = true
metrics = false
logs = false
trace_sample_rate = 0.5

Environment Variable Override

GORDON_TELEMETRY_ENABLED=true
GORDON_TELEMETRY_ENDPOINT=http://localhost:5080/api/default
GORDON_TELEMETRY_AUTH_TOKEN=YWRtaW5AZXhhbXBsZS5jb206c2VjcmV0
GORDON_TELEMETRY_TRACES=true
GORDON_TELEMETRY_METRICS=true
GORDON_TELEMETRY_LOGS=true
GORDON_TELEMETRY_TRACE_SAMPLE_RATE=1.0