Skip to content

Wiring metrics

Prometheus-backed panels read the selected cluster's prometheus_endpoint. The endpoint must be reachable from the control plane. Provider adapters may supply scoped fallbacks, but configuring an endpoint alone does not establish workload ownership, instrumentation, collector completeness or healthy usage. See workload signals for each signal's source, scope and required identity evidence.

What to install

Astrolift does not deploy Prometheus. Install one in the tenant cluster; kube-prometheus-stack is what the platform is developed against. Leave its default exporters on. The panels are built on series those exporters already produce:

Panel Series Comes from
Cluster saturation, node and pod counts kube_node_*, kube_pod_*, kube_deployment_* kube-state-metrics
Workload and pod CPU / memory container_cpu_usage_seconds_total, container_memory_working_set_bytes cAdvisor
Traffic, errors, latency ingress-controller or application metrics see the app side

kube-prometheus-stack is a toggle of the astrolift-prereqs umbrella chart that astro cluster bootstrap installs: the k8s overlay enables it, the aws, gcp and azure overlays leave it off (those clouds default to the native backend). Turn it on with a --values-file that sets kube-prometheus-stack.enabled: true, or install the chart yourself. Ensure Prometheus selects the workload's PodMonitor or ServiceMonitor and its namespace. Creating a monitor does not by itself guarantee a scrape; review the installation's actual selectors and collector labels.

Success check, from inside the cluster:

kubectl -n <monitoring-namespace> get prometheus,servicemonitor,podmonitor
kubectl -n <monitoring-namespace> get pods -l app.kubernetes.io/name=prometheus

<monitoring-namespace> is the namespace you installed the stack into (astrolift-system when it came from astro cluster bootstrap). The Prometheus pod is Running and the monitors for your workload namespace are listed.

The reachability contract

Queries originate from the control plane's network. When it is outside the workload cluster, ordinary cluster-local addresses are not sufficient:

  • a Service ClusterIP is not routable; it is an iptables construct inside the cluster's network namespace;
  • the in-cluster DNS name that resolves to it (prometheus.monitoring.svc.cluster.local) is not resolvable either.

The natural first attempt is exactly that name, and it fails with a connection error that looks like Prometheus is down. Astrolift recognises an in-cluster address and says so rather than reporting it unreachable, but the fix is on your side: put an internal load balancer in front of the Prometheus service, reachable from the control plane's network, and point the endpoint at that.

Pod IPs are a different case. On EKS they are real VPC addresses and route from the control plane, which is why auto-discovery works, but a pod IP changes when the pod is rescheduled.

Setting the endpoint

Three write paths, in increasing order of durability:

Auto-discovery. Run Refresh cluster management from the cluster's Settings tab (or bringClusterIntoManagement on first adoption). The capability probe finds the Prometheus pod and records its IP in the cluster's capabilities. Nothing to configure, but it is a pod IP: it goes stale on the next reschedule and needs another refresh.

The dashboard. Set prometheus_endpoint in the cluster's provider config on the cluster Settings tab. Survives everything; set by hand.

The registration command. For an install that manages clusters declaratively:

python manage.py register_tenant_cluster \
  --slug <cluster-slug> \
  --prometheus-endpoint http://<internal-load-balancer>:9090

<cluster-slug> is the registered cluster's slug (re-running with the same slug updates the row); <internal-load-balancer> is the address of the load balancer you put in front of the Prometheus service. Or, equivalently, set ASTROLIFT_CLUSTER_PROMETHEUS_ENDPOINT in the control plane's environment: the command runs on every container start and re-asserts it. Omitting both leaves whatever the row already carries, including a probe-discovered value.

A value set in provider config wins over the probe-discovered one.

Success check: the cluster's metrics panels render, and from the control plane's network curl -s http://<internal-load-balancer>:9090/-/ready returns Prometheus Server is Ready. If the panel reports the endpoint as cluster-internal, the value ends in .svc, .svc.cluster.local or .cluster.local, or is a loopback address; replace it with the load balancer address.

The app side

Cluster panels need the matching exporter series as well as an endpoint. Physical workload CPU/memory needs verified container identities and limits. Environment-level traffic, errors and latency can use ingress metrics; exact workload request metrics require application instrumentation and canonical GUID labels. These sources have different scopes. That contract, and the [workloads.metrics] table that declares it, is documented in the astrolift.toml reference.

Measurement scope and workload identity

See workload signals before treating a chart as workload evidence. Namespace ingress and resource aggregates are app-environment observations. Workload request metrics require canonical GUID instrumentation labels and verified runtime membership; physical CPU/memory measurements also require collector identities and matching limits. Missing samples, permissions, collectors or limits are unavailable rather than zero.