Operator Runbooks¶
Start with the resource's dashboard detail and the control-plane audit/event records. Use kubectl only after you have identified the cluster and namespace Astrolift recorded for that resource.
Cluster is registered but not managed¶
Symptoms: the cluster appears in Admin → Clusters as registered, app registration refuses to continue, or management reports a prerequisite/authentication error.
Registration stores connection metadata and probes capabilities. It does not make the cluster managed. The normal sequence is:
- Register the cluster.
- Install or reconcile prerequisites.
- Bring the cluster into management.
- Install the keep-alive agent credential and Deployment.
Render the prerequisite Helm operation before applying it:
astro cluster bootstrap --cluster-slug <slug> --dry-run
astro cluster bootstrap --cluster-slug <slug> --wait-timeout 10m
If the release already exists, add --upgrade. For a custom values overlay, add --values-file <path>.
Authentication failures¶
The registration dialog supports kubeconfig, service_account_token, and exec_plugin. Test the same identity and endpoint Astrolift uses:
kubectl --kubeconfig <file> auth can-i get namespaces
kubectl --kubeconfig <file> auth can-i create deployments --all-namespaces
For an exec_plugin cloud cluster, refresh the operator kubeconfig before running bootstrap:
aws eks update-kubeconfig --name <cluster> --region <region>
# or the corresponding gcloud/az command
Do not paste a short-lived personal credential into a production cluster registration. Prefer workload identity or a purpose-built service account.
Missing prerequisites¶
Check controllers and CRDs directly:
kubectl get crd certificates.cert-manager.io
kubectl get pods -A | grep -E 'cert-manager|ingress|external-dns'
helm list -A
Use the cloud overlay selected by astro cluster bootstrap; a generic hand-written chart install can omit the provider-specific requirements the control plane later expects.
Keep-alive agent never reports¶
The raw agent key is returned once and must land in the cluster Secret. Use the CLI so it is written without being printed:
astro operator cluster install-agent --slug <slug>
astro operator cluster deploy-agent --slug <slug>
kubectl -n astrolift-system get deploy,pod,secret | grep astrolift-agent
If the Deployment reports CreateContainerConfigError, confirm the astrolift-agent Secret exists in astrolift-system.
Deployment fails or remains in flight¶
Open /deployments/<id> and record the app, environment, cluster, image reference, and failed lifecycle step. The application namespace is normally <org-slug>-<app-slug>, but use the namespace stored on the app when it differs.
Image pull failure¶
Check the rendered image reference, registry reachability, pull-secret/workload-identity wiring, and whether the immutable tag or digest actually exists. Re-running a deployment cannot fix an image that was never published.
Crash or readiness failure¶
kubectl -n <namespace> logs <pod> --all-containers --previous
kubectl -n <namespace> describe pod <pod>
astro app logs --app <slug> --since 30m
Compare the container port and readiness configuration with the process actually listening in the image. Confirm required app or managed-service secrets are materialized before retrying.
Pending pods or rollout timeout¶
kubectl -n <namespace> get pods
kubectl -n <namespace> describe pod <pending-pod>
kubectl get nodes
kubectl describe node
Typical causes are insufficient CPU/memory, an unavailable storage class, a node selector with no match, taints without tolerations, or a minimum replica count the cluster cannot schedule.
DNS or certificate failure¶
kubectl -n <namespace> get ingress,certificate,challenge
kubectl -n <namespace> describe certificate <name>
dig +short <hostname>
Use the hostname and challenge records shown by Astrolift. Base-domain, custom-domain, HTTP-01, and DNS-01 paths vary by provider; do not assume every app uses a wildcard record.
Cancel a runaway agent¶
List active tasks, inspect the target, then cancel its task ID:
For a provisioning or running Kubernetes task, cancellation deletes the Kubernetes Job and pod before recording cancelled. It also removes the per-task Secret. Treat an error from cancel as a failed hard stop, not a successful state update.
Verify both planes:
Every Kubernetes agent Job also receives activeDeadlineSeconds from the task timeout as a backstop. Set a bounded timeout when dispatching:
If a cancelled task returns or duplicates, inspect the workflow/trigger concurrency policy. Cancelling the pod alone does not disable a cron, loop, source webhook, or repeated workflow trigger.
Agent logs show only VNC infrastructure¶
First distinguish the container streams. VNC-enabled images can have Xvfb/x11vnc output in one container while the agent runtime logs from another.
astro agent inspect <task-id>
astro agent logs <task-id> --help
kubectl -n <task-namespace> get pod -l astrolift.dev/task-id=<task-id> -o jsonpath='{.items[0].spec.containers[*].name}'
kubectl -n <task-namespace> logs <pod> -c <agent-container>
The control plane log resolver uses the namespace frozen on the task and the task label. If direct kubectl logs contains application output but astro agent logs does not, record the task ID, namespace, pod, container, and time range; that indicates a collection/routing defect rather than an application silence.
Interactive terminal output is not automatically equivalent to process stdout/stderr. A program attached to a PTY may buffer or route output differently from the container log stream.
Agent callback or findings are missing¶
Inspect the task's rendered environment without printing secret values. Agent Jobs must receive an absolute AGENT_CALLBACK_URL and a task-scoped AGENT_CALLBACK_TOKEN. The worker/API environment must have an absolute PLATFORM_API_URL; spawn fails if the callback URL cannot be constructed.
The callback endpoint is:
The task-scoped token is bound to that task and cleared at terminal state. A reporting client must treat non-2xx responses as failure; a local helper that silently no-ops is not evidence the platform received findings. Confirm the task's callback interaction/audit record and terminal result in Astrolift.
Teardown is stuck¶
Preview deregistration before starting destructive work:
The confirmed path starts an asynchronous workflow with an approximately five-minute cancellation grace period. After destructive work begins, teardown is monotonic; re-run it to reconcile partial failure rather than trying to roll it back.
Namespace remains terminating¶
kubectl get namespace <namespace> -o json | jq '.status.conditions'
kubectl api-resources --verbs=list --namespaced -o name \
| xargs -n1 kubectl get -n <namespace> --ignore-not-found
Remove a finalizer only after proving its external resource is already gone or deliberately retained. Manual namespace deletion before the deregister workflow bypasses ordered cleanup of domains, managed services, secrets, identity, webhooks, and registry resources.
Volume or cloud data remains¶
Data retention and force-destroy are independent controls. The safe default can retain snapshots, buckets, or volumes. Read the teardown inventory/report before treating a retained resource as a leak.
Rotate secrets and tokens¶
App or agent secret¶
Use the resource's Secrets page or CLI command. A rotation is a new audited value write; it does not imply that every running pod has restarted. Verify the subsequent deployment/restart before revoking the upstream credential.
Reveal is possible only for providers that support value disclosure and callers with the read permission and token scope. Write-only backends cannot reveal a value after submission.
Deploy token¶
Create or rotate from the app's Tokens page, copy the one-time plaintext into CI, verify a build with the new token, then revoke the old credential. The default rotate policy provides a 24-hour overlap; revoke is immediate.
Webhook secret¶
Rotate from the webhook detail, deploy the new receiver secret, and accept the previous secret only for the configured grace period (one hour by default, bounded to 24 hours). Test a signed delivery from the dashboard before removing the old secret.
Provider-specific checks¶
AWS ALB reports FailedNetworkReconcile¶
Inspect the AWS Load Balancer Controller flags and the security groups tagged for the cluster:
kubectl -n kube-system get deploy aws-load-balancer-controller -o yaml \
| grep -i -E 'manage-backend|backend-security-group'
aws ec2 describe-security-groups --region <region> \
--filters 'Name=tag:kubernetes.io/cluster/<name>,Values=owned' \
--query 'SecurityGroups[].GroupId'
Two security groups tagged owned can make backend security-group selection ambiguous. Astrolift's EKS driver pins backend security-group behavior when it can resolve the intended group; do not add a broad ingress rule before confirming the actual controller configuration.
Managed Bedrock model is listed but invocation fails as legacy¶
AWS can continue listing an inference profile that Bedrock refuses to invoke after inactivity. Test invocation, not catalogue presence. An explicit provider message that the model is Legacy is a model-lifecycle failure, not an IAM failure.
Verify a replacement model from the same identity and region before changing ANTHROPIC_MODEL. Keep the model ID in deploy-time configuration so the next retirement does not require an application release.
EKS or GKE nodes do not scale¶
Start with pending-pod scheduling events, then inspect the cloud autoscaler/node-pool health. Check capacity availability, current images, quota, subnets/IPs, and node identity. A healthy autoscaler cannot overcome an impossible selector or a missing instance profile.