Operator Runbooks¶

Start with the resource's dashboard detail and the control-plane audit/event records. Use kubectl only after you have identified the cluster and namespace Astrolift recorded for that resource.

Cluster is registered but not managed¶

Symptoms: the cluster appears in Admin → Clusters as registered, app registration refuses to continue, or management reports a prerequisite/authentication error.

Registration stores connection metadata and probes capabilities. It does not make the cluster managed. The normal sequence is:

Register the cluster.
Install or reconcile prerequisites.
Bring the cluster into management.
Install the keep-alive agent credential and Deployment.

Render the prerequisite Helm operation before applying it:

astro cluster bootstrap --cluster-slug <slug> --dry-run
astro cluster bootstrap --cluster-slug <slug> --wait-timeout 10m

<slug> is the cluster slug shown in Clusters. If the release already exists, add --upgrade. For a custom values overlay, add --values-file <path>. Success check: helm -n astrolift-system list shows astrolift-prereqs as deployed and kubectl -n astrolift-system get pods shows the controllers Running.

Authentication failures¶

The registration dialog supports kubeconfig, service_account_token, and exec_plugin. Test the same identity and endpoint Astrolift uses:

kubectl --kubeconfig <file> auth can-i get namespaces
kubectl --kubeconfig <file> auth can-i create deployments --all-namespaces

For an exec_plugin cloud cluster, refresh the operator kubeconfig before running bootstrap:

aws eks update-kubeconfig --name <cluster> --region <region>
# or the corresponding gcloud/az command

Do not paste a short-lived personal credential into a production cluster registration. Prefer workload identity or a purpose-built service account.

Missing prerequisites¶

Check controllers and CRDs directly:

kubectl get crd certificates.cert-manager.io
kubectl get pods -A | grep -E 'cert-manager|ingress|external-dns'
helm list -A

Use the cloud overlay selected by astro cluster bootstrap; a generic hand-written chart install can omit the provider-specific requirements the control plane later expects.

Keep-alive agent never reports¶

The agent is installed through the control plane: a server-side workflow writes the credential and Deployment over the control plane's network, and the current key keeps working until an authenticated heartbeat confirms the replacement. No raw key or local kubeconfig is involved. The contract, including recovery after a lost reply, is in Install the cluster keep-alive agent.

astro operator cluster install-agent --slug <slug> --request-file agent-install.json
astro operator cluster agent-install-status --install-id <install-id>
astro operator cluster deploy-agent --slug <slug>
kubectl -n astrolift-system get deploy,pod,secret | grep astrolift-agent

<install-id> is the installation UUID install-agent prints; agent-install.json is the private request file, keep it and retry with the same file after a lost reply. Success is agent-install-status reporting a confirmed heartbeat, not the request being accepted. The commands need cluster.manage, which the default sign-in does not carry: sign in with astro auth login --scope clusters.

deploy-agent refuses to run until the astrolift-agent Secret exists in astrolift-system; if the Deployment reports CreateContainerConfigError, that Secret is missing or stale, so run install-agent again. The log collector is a separate install, see Cluster log collector.

Deployment fails or remains in flight¶

Open /deployments/<id> and record the app, environment, cluster, image reference, and failed lifecycle step. The application namespace is normally <org-slug>-<app-slug>, but use the namespace stored on the app when it differs. In the commands below <namespace> is that namespace and <slug> is the app slug.

Image pull failure¶

kubectl -n <namespace> get pods
kubectl -n <namespace> describe pod <pod>

Check the rendered image reference, registry reachability, pull-secret/workload-identity wiring, and whether the immutable tag or digest actually exists. Re-running a deployment cannot fix an image that was never published.

Crash or readiness failure¶

kubectl -n <namespace> logs <pod> --all-containers --previous
kubectl -n <namespace> describe pod <pod>
astro app logs --app <slug> --since 30m

Compare the container port and readiness configuration with the process actually listening in the image. Confirm required app or managed-service secrets are materialized before retrying.

Pending pods or rollout timeout¶

kubectl -n <namespace> get pods
kubectl -n <namespace> describe pod <pending-pod>
kubectl get nodes
kubectl describe node

Typical causes are insufficient CPU/memory, an unavailable storage class, a node selector with no match, taints without tolerations, or a minimum replica count the cluster cannot schedule.

DNS or certificate failure¶

kubectl -n <namespace> get ingress,certificate,challenge
kubectl -n <namespace> describe certificate <name>
dig +short <hostname>

Use the hostname and challenge records shown by Astrolift. Base-domain, custom-domain, HTTP-01, and DNS-01 paths vary by provider; do not assume every app uses a wildcard record.

App runs but its host answers the ingress default 404¶

Signature. The deployment reads running and the pods are healthy, but kubectl -n <namespace> get ingress lists nothing for the host, the public URL answers the ingress controller's default 404, and the app header reads Needs reprovision or No managed domain instead of Live. The URL card shows the bare subdomain with no zone. This is what every app registered while the platform default managed zone was missing (deleted, or never created) looks like; the registration had no zone to claim a hostname in, so no Ingress was rendered.

Fix. Check the zone first: astro operator domains list (or Admin, Domains) must show a live platform default. Restore or recreate it, then run the managed-domain binding backfill from the Domains screen; it binds every environment that has no zone and sets its URL. Then, per affected app, astro app reprovision --app <slug> --wait, or the Reprovision button on the URL card. Nothing is rebuilt, deleted or restarted unless the rendered workload changed. Do not use Force redeploy for this; it recreates the environment and drops traffic.

Verify. astro app show --app <slug> reads reachable; kubectl -n <namespace> get ingress,certificate shows the host's Ingress and a Ready certificate; curl -sI https://<host>/ answers with the app or a 302 to the install's sign-in. The Apps list filtered on Needs reprovision is empty.

Domain DNS health¶

Start in Domains, open the zone, and use its Diagnostics tab before reaching for dig or the cloud console. Domains and DNS explains the stages and tools; this section covers the operator-only provider reads. Zone inventory, Route 53 binding and the zone certificate are platform-operator reads. A tenant sees PLATFORM_OPERATOR_REQUIRED and the provider is never called.

Read the reason code¶

Each check shows its state (OK, MISMATCH, UNKNOWN, ERROR or UNSUPPORTED), a reason code and an explanation with the next step. Common codes:

Reason	Meaning	Next step
`DNS_NXDOMAIN`	The name does not exist; a lookup reports `MISMATCH`, never healthy	Check the record name in the zone, and that public delegation points at the provider zone
`DNS_TIMEOUT`	The resolver never answered; `ERROR` in probes, `UNKNOWN` in the public-delegation check	Retry; a timeout does not show whether the record exists
`PUBLIC_DELEGATION_MISMATCH`	Public nameservers differ from the provider zone's	Review the registrar delegation; see public delegation
`PROVIDER_ACCESS_DENIED`	The provider refused the read	Grant the connection read access to the zone, then refresh
`PROVIDER_ZONE_NOT_BOUND`	No provider zone is bound to this domain	Bind the zone through the connection wizard
`PROVIDER_CLUSTER_UNAVAILABLE`	The bound cluster is missing, inactive or foreign, or its AWS account changed	Bind the zone again from Connect DNS
`TOOL_NOT_INSTALLED`	The probe utility is not available on the control-plane host	Use another tool; the check is not reported as passing

On the Routing view, Check HTTPS runs a live HTTPS/TLS check for an app route's hostname. From the CLI, astro operator domains verify <zone> checks the published TXT proof of control and astro operator domains probe / lookup run the same resolver checks.

Bind a Route 53 hosted zone through a cluster¶

Open the domain's detail and start Connect DNS, then choose Route53. With a reviewed domain, the step lists hosted zones through a cluster's AWS credential, the identity that writes DNS for that cluster:

Cluster credential: pick one of the organization's AWS clusters, or a shared one, that is active and has a verified account. Another organization's cluster is not offered.
Zones: the hosted zones that credential can read are listed, private zones marked. The list stops at 500 zones and says so.
Bind to <domain>: only the zone named exactly like the domain can be bound. The binding is platform-written and pins the AWS account the zone was seen in.

From then on, diagnostics read that zone's records with the cluster's credential only (bindingSource: CLUSTER_CREDENTIAL). If the cluster becomes unavailable or its account changes, the provider zone reports PROVIDER_CLUSTER_UNAVAILABLE; it never falls back to the control plane's own identity. The API operations are route53DnsClusters, route53DnsZones(input: { clusterId }) and attachRoute53DnsZone(input: { domainId, expectedDomainVersion, clusterId, zoneId, zoneName }).

Check the zone certificate¶

The Diagnostics tab shows a Zone certificate panel under the provider zone: state, explanation, provider status, validation record count and check time. It reads the zone's recorded certificate with the provisioning cluster's account-pinned ACM client and checks that the certificate covers the zone.

Certificate	Reason	State
Issued	`CERTIFICATE_ISSUED`	`OK`
Waiting for DNS validation	`CERTIFICATE_PENDING_VALIDATION`	`UNKNOWN`
Failed, timed out, expired, revoked or inactive	`CERTIFICATE_<status>`	`ERROR`
Host mismatch, changed account or another provider refusal	the provider code, such as `NATIVE_CERTIFICATE_HOST_MISMATCH` or `NATIVE_ACCOUNT_CHANGED`	`ERROR`
Unreadable	`CERTIFICATE_UNAVAILABLE`	`ERROR`
No certificate recorded	`CERTIFICATE_NOT_RECORDED`	`UNSUPPORTED`
No verified AWS cluster	`CERTIFICATE_CLUSTER_UNAVAILABLE`	`UNKNOWN`

A pending certificate needs its validation records to resolve in the public zone; check delegation first. Through the API, read certificate { state reason checkedAt status validationRecords } on astroliftManagedDomainDiagnostics.

Cancel a runaway agent¶

List active tasks, inspect the target, then cancel its task ID:

astro agent ls
astro agent inspect <task-id>
astro agent cancel <task-id> --yes

<task-id> is the task GUID from astro agent ls. For a provisioning or running Kubernetes task, cancellation deletes the Kubernetes Job and pod before recording cancelled. Treat an error from cancel as a failed hard stop, not a successful state update.

Verify both planes:

astro agent inspect <task-id>
kubectl -n <task-namespace> get job,pod -l astrolift.dev/task-id=<task-id>

<task-namespace> is the namespace astro agent inspect reports for the task. Success is inspect showing cancelled and the kubectl listing empty.

Every Kubernetes agent Job also receives activeDeadlineSeconds from the task timeout as a backstop. Set a bounded timeout when dispatching:

astro agent dispatch <agent-slug> --timeout 900 --tail

If a cancelled task returns or duplicates, inspect the workflow/trigger concurrency policy. Cancelling the pod alone does not disable a cron, loop, source webhook, or repeated workflow trigger.

Agent logs show only VNC infrastructure¶

First distinguish the container streams. VNC-enabled images can have Xvfb/x11vnc output in one container while the agent runtime logs from another.

astro agent inspect <task-id>
astro agent logs <task-id> --help
kubectl -n <task-namespace> get pod -l astrolift.dev/task-id=<task-id> -o jsonpath='{.items[0].spec.containers[*].name}'
kubectl -n <task-namespace> logs <pod> -c <agent-container>

The control plane log resolver uses the namespace frozen on the task and the task label. If direct kubectl logs contains application output but astro agent logs does not, record the task ID, namespace, pod, container, and time range; that indicates a collection/routing defect rather than an application silence.

Interactive terminal output is not automatically equivalent to process stdout/stderr. A program attached to a PTY may buffer or route output differently from the container log stream.

Agent callback or findings are missing¶

Inspect the task's rendered environment without printing secret values. Agent Jobs must receive an absolute AGENT_CALLBACK_URL and a task-scoped callback key in ASTROLIFT_CLUSTER_KEY (prefix alft_cb_, sent as Authorization: Bearer). The worker/API environment must have an absolute PLATFORM_API_URL; spawn fails with PLATFORM_API_URL must be an absolute HTTP(S) URL for agent callbacks if the callback URL cannot be constructed.

The callback endpoint is:

/api/dispatch/v1/agents/<task-guid>/callback/

The task-scoped key is bound to that task, rotated on every spawn and cleared at terminal state; a stale key gets 404 task not found or callback token expired. A reporting client must treat non-2xx responses as failure; a local helper that silently no-ops is not evidence the platform received findings. Confirm the task's callback interaction/audit record and terminal result in Astrolift (astro agent inspect <task-id> shows the terminal result payload). The callback contract is documented in Agent completion callbacks.

Teardown is stuck¶

Preview deregistration before starting destructive work:

astro app deregister <app-slug>
astro app deregister <app-slug> --yes

Without --yes only the teardown inventory is printed and nothing changes. The confirmed path starts an asynchronous workflow with an approximately five-minute cancellation grace period, cancellable with the printed workflow id through the cancelAstroliftDeregister mutation. After destructive work begins, teardown is monotonic; re-run it to reconcile partial failure rather than trying to roll it back. Success check: astro app show <app-slug> no longer finds the app and kubectl get namespace <namespace> returns NotFound.

Namespace remains terminating¶

kubectl get namespace <namespace> -o json | jq '.status.conditions'
kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n1 kubectl get -n <namespace> --ignore-not-found

Remove a finalizer only after proving its external resource is already gone or deliberately retained. Manual namespace deletion before the deregister workflow bypasses ordered cleanup of domains, managed services, secrets, identity, webhooks, and registry resources.

Volume or cloud data remains¶

Data retention and force-destroy are independent controls. The safe default can retain snapshots, buckets, or volumes. Read the teardown inventory/report before treating a retained resource as a leak.

Rotate secrets and tokens¶

App or agent secret¶

Use the resource's Secrets page or CLI command. A rotation is a new audited value write; it does not imply that every running pod has restarted. Verify the subsequent deployment/restart before revoking the upstream credential.

Reveal is possible only for providers that support value disclosure and callers with the read permission and token scope. Write-only backends cannot reveal a value after submission.

Deploy token¶

Create or rotate from the app's Tokens page, copy the one-time plaintext into CI, verify a build with the new token, then revoke the old credential. The default rotate policy provides a 24-hour overlap; revoke is immediate.

Webhook secret¶

Rotate from the webhook detail, deploy the new receiver secret, and accept the previous secret only for the configured grace period (one hour by default, bounded to 24 hours). Test a signed delivery from the dashboard before removing the old secret.

Provider-specific checks¶

AWS ALB reports `FailedNetworkReconcile`¶

Inspect the AWS Load Balancer Controller flags and the security groups tagged for the cluster:

kubectl -n kube-system get deploy aws-load-balancer-controller -o yaml \
  | grep -i -E 'manage-backend|backend-security-group'
aws ec2 describe-security-groups --region <region> \
  --filters 'Name=tag:kubernetes.io/cluster/<name>,Values=owned' \
  --query 'SecurityGroups[].GroupId'

Two security groups tagged owned can make backend security-group selection ambiguous. Astrolift's EKS driver pins backend security-group behavior when it can resolve the intended group; do not add a broad ingress rule before confirming the actual controller configuration.

Managed Bedrock model is listed but invocation fails¶

AWS can continue listing an inference profile that Bedrock refuses to invoke. Test invocation, not catalogue presence; a model-lifecycle refusal is not an IAM failure.

Verify a replacement model from the same identity and region before changing the model. The per-cluster override is provider_config["bedrock_model_id"] (and bedrock_small_fast_model_id) on the cluster; the process-wide fallback is ANTHROPIC_MODEL / ANTHROPIC_SMALL_FAST_MODEL in the control plane's environment. Keep the model ID in that configuration so the next retirement does not require an application release.

EKS or GKE nodes do not scale¶

Start with pending-pod scheduling events, then inspect the cloud autoscaler/node-pool health. Check capacity availability, current images, quota, subnets/IPs, and node identity. A healthy autoscaler cannot overcome an impossible selector or a missing instance profile.