Skip to content

Operator Runbooks

Recovery procedures for the failure modes you'll actually hit operating an Astrolift install. Each entry is Symptom → Root cause → Resolution → Prevention.

These cover the cluster lifecycle, the deploy loop, teardown, secret rotation, and the provider-specific traps on AWS and GCP. For the happy path, see the Quickstart. For the full install, see Installing Astrolift.

Where to look first

Most deploy and cluster issues surface in two places: the Deployments detail page lifecycle log (/deployments/<id>) and the cluster row Health check (/clusters). Read those before reaching for kubectl — the platform usually already named the failing step.

1. Cluster registration stuck / failed

The cluster never reaches a healthy state after Clusters → Register, or the Health check button reports a failure.

1a. Missing IAM role or kubeconfig permissions

Symptom. Registration hangs on "verifying connectivity", or the health check returns forbidden / Unauthorized / could not list namespaces.

Root cause. The credential the control plane uses can't reach or can't authorize against the cluster API. For the IAM path, the role isn't assumable (trust policy) or lacks EKS describe/access-entry permission. For the kubeconfig path, the embedded service account lacks RBAC for the namespaces Astrolift manages.

Resolution.

  • IAM (EKS): confirm the control plane's identity can assume the role and that the role has an EKS access entry (or aws-auth mapping) granting cluster access: 
    aws sts assume-role --role-arn <arn> --role-session-name astro-check
aws eks describe-cluster --name <cluster> --region <region>
aws eks list-access-entries --cluster-name <cluster> --region <region>
  • kubeconfig: test the exact kubeconfig you pasted, then verify RBAC: 
    kubectl --kubeconfig pasted.yaml auth can-i create deployments -n default
kubectl --kubeconfig pasted.yaml get ns
    Grant the service account a ClusterRole that covers namespaces, deployments, services, ingresses, secrets, and pods/exec.

Prevention. Register with a purpose-built service account (or IRSA role), not a personal admin context that can rotate or expire. Run the Health check immediately after registering and before the first deploy.

1b. cert-manager not installed on the target cluster

Symptom. Health check flags cert-manager not detected. Or registration succeeds but the first app deploy stalls at the TLS step and no Certificate resource ever goes Ready.

Root cause. Astrolift delegates TLS issuance to cert-manager on the tenant cluster. Without the CRDs and controller, certificate requests are never reconciled.

Resolution.

kubectl get crd certificates.cert-manager.io   # missing -> not installed
helm repo add jetstack https://charts.jetstack.io && helm repo update
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace --set crds.enabled=true
kubectl -n cert-manager rollout status deploy/cert-manager-webhook

Re-run the cluster health check.

Prevention. Make cert-manager a registration prerequisite (it's listed in the Quickstart prerequisites). The astrolift-prereqs umbrella chart in astrolift-opscode installs it alongside the other cluster operators if you'd rather manage it there.

1c. Network policy blocking control plane → cluster communication

Symptom. Connectivity is intermittent or one-directional: the health check passes but heartbeats later drop, or vice versa. kubectl from your laptop works fine, but the control plane can't reach the API server or the in-cluster agent.

Root cause. A deny-by-default NetworkPolicy (or a security group / firewall rule) is blocking the control-plane source from the cluster API endpoint or the namespaces it manages. This is common when the control plane runs outside the tenant cluster.

Resolution.

  • Confirm the cluster API endpoint is reachable from the control plane's egress IP/SG (private-endpoint EKS clusters need the control plane in an allowed CIDR).
  • Audit NetworkPolicies in the managed namespaces — Astrolift's own policies are deny-by-default; an over-broad operator policy can also block the platform's agent: 
    kubectl get networkpolicy -A
  • Allow ingress from the control-plane source to the platform agent and egress to the cluster API.

Prevention. Document the control plane's egress IP/SG and add it to the cluster's API allowlist at registration time. Keep platform-managed namespaces' network policies under Astrolift's control rather than a cluster-wide operator policy that doesn't know about them.

2. App deploy stuck in "deploying"

The deployment sits in deploying past the normal ~2–3 minutes. Open /deployments/<id> and read the lifecycle log — it names the step that's wedged. Match it below.

2a. Image pull error

Symptom. Lifecycle log stalls after "apply"; pods show ImagePullBackOff / ErrImagePull.

Root cause. Expired or missing registry credentials, or a wrong image tag/digest (the rendered manifest references an image that was never pushed, or a tag that was overwritten).

Resolution.

kubectl -n <org>-<app> describe pod <pod> | grep -A5 -i "failed to pull"
kubectl -n <org>-<app> get serviceaccount default -o yaml   # imagePullSecrets present?
  • Wrong tag: redeploy from the Deployments page (Redeploy) so the platform re-renders against the image it actually pushed.
  • Expired creds: the registry pull secret rotates with the cluster's workload identity; re-run the cluster Health check to refresh it, or for ECR confirm the IRSA role still has ecr:GetAuthorizationToken.

Prevention. Use immutable digests, not mutable :latest-style tags, so a tag overwrite can't desync the manifest from the registry. Let the platform manage pull secrets via workload identity rather than hand-placed imagePullSecrets.

2b. DNS not propagating / cert challenge failing

Symptom. Deploy reaches "wait healthy" / "assign URL" but the URL returns TLS errors or never resolves. The Certificate is pending; the ACME Challenge is failing.

Root cause. The cert-manager HTTP-01/DNS-01 challenge can't be satisfied — usually because base_domain is wrong (the hostname doesn't point at this install's ingress) or DNS hasn't propagated.

Resolution.

kubectl -n <org>-<app> get certificate,challenge
kubectl -n <org>-<app> describe challenge <name>   # shows why ACME is failing
dig +short my-first-app.astro.example.com           # resolves to the ingress?
  • Confirm the wildcard/app hostname resolves to this install's ingress address.
  • Confirm the control plane's baseDomain / ASTROLIFT_BASE_DOMAIN matches the zone you delegated.
  • For custom domains, the validation TXT token must be in DNS — copy it from /apps/<slug>/domains and click Recheck after it propagates.

Prevention. Verify the base zone delegation and a wildcard record during install, before the first app. Set baseDomain to a zone you actually control — it's a required value precisely so it can't silently default to something wrong.

2c. Pod CrashLoopBackOff

Symptom. Lifecycle log shows the rollout applied but never goes healthy; pods cycle CrashLoopBackOff.

Root cause. The app process exits on start — a bad/missing env var, a secret that isn't attached, or a healthcheck path/port that doesn't match what the process serves (the platform marks the pod unready and the rollout never completes).

Resolution.

kubectl -n <org>-<app> logs <pod> --previous   # why the last start died
kubectl -n <org>-<app> describe pod <pod>       # readiness probe failures
  • Missing/empty env: add it on /apps/<slug>/secrets, or confirm an attached managed service is active so its envelope (e.g. DATABASE_URL) is injected.
  • Healthcheck mismatch: the healthcheck in astrolift.toml (e.g. http:/health) must hit a path/port the container actually answers 200 on. Fix it in the manifest and redeploy.

Prevention. Keep the manifest's port and healthcheck in sync with the app. Use the Config page render preview to catch obvious manifest errors before deploy. Attach secrets/services before the deploy that depends on them.

2d. Rollout timeout

Symptom. Lifecycle log ends in a rollout timeout; some replicas never become Ready even though the image is fine and the app starts.

Root cause. Insufficient cluster resources to schedule the requested replicas (CPU/memory requests exceed allocatable, or no node fits), or a misconfigured HPA whose minimum can't be satisfied. Pods sit Pending.

Resolution.

kubectl -n <org>-<app> get pods            # any Pending?
kubectl -n <org>-<app> describe pod <pod>  # "Insufficient cpu/memory" / "didn't match"
kubectl describe node | grep -A5 Allocated
kubectl -n <org>-<app> get hpa
  • Resource pressure: scale the node group out (see §7 for EKS) or lower the workload's requests/replica count in astrolift.toml.
  • HPA: confirm min/max are sane and the metrics server is running; an HPA min above what the cluster can schedule will time out forever.

Prevention. Right-size requests/limits and HPA bounds per workload on /apps/<slug>/workloads/<slug>. Keep headroom on the node group, or enable the cluster autoscaler so a larger replica count can actually land.

3. ALB FailedNetworkReconcile (AWS-specific)

Symptom. On EKS, the ALB target group shows targets unhealthy after an Ingress change — classically right after toggling the auth gate on an app. The AWS Load Balancer Controller logs a FailedNetworkReconcile loop: expected exactly one securityGroup tagged with kubernetes.io/cluster/....

Root cause. A dual-SG tag conflict. Both the EKS cluster security group and the node shared security group carry kubernetes.io/cluster/<name>: owned. The controller's TargetGroupBinding reconciler discovers the backend SG by that tag and expects exactly one; finding two, it can't decide which to add the node-traffic rule to and loops, leaving target-group health stuck on every Ingress reconcile.

Resolution.

  1. Confirm the controller is told not to manage the backend SG rules itself — it should run with manageBackendSecurityGroupRules=false and an explicit backendSecurityGroup. Astrolift sets both at cluster bootstrap (see backend/providers/aws/cluster_eks.py, where the ALB Helm values pin manageBackendSecurityGroupRules to False once a backendSecurityGroup is discovered or supplied): 
    kubectl -n kube-system get deploy aws-load-balancer-controller -o yaml \
  | grep -i -E "manage-backend|backend-security-group"
  2. With the controller no longer managing the rule, the inbound rule must exist once. If targets are still unhealthy, add inbound TCP:80 (and your target port) from the node SG to the backend SG manually: 
    aws ec2 authorize-security-group-ingress \
  --group-id <backend-sg> --protocol tcp --port 80 \
  --source-group <node-sg> --region <region>
  3. Confirm the dual-tag situation: 
    aws ec2 describe-security-groups --region <region> \
  --filters "Name=tag:kubernetes.io/cluster/<name>,Values=owned" \
  --query 'SecurityGroups[].GroupId'
# two IDs back == the conflict

Prevention. Set manageBackendSecurityGroupRules=false and pin backendSecurityGroup at cluster registration time — Astrolift's EKS registration does this for you, which is why clusters registered through the platform don't hit the loop. If you install the controller by hand outside Astrolift, replicate those two values. Don't tag both the cluster SG and the node shared SG owned; one owned, others shared.

4. Teardown stuck

TeardownAppWorkflow (or a cluster teardown) doesn't finish; resources linger.

4a. Namespace stuck in Terminating

Symptom. The app's namespace sits in Terminating indefinitely; the teardown report never closes out.

Root cause. A finalizer on a resource in the namespace can't complete — commonly a cert-manager, load-balancer, or CRD finalizer waiting on an upstream cloud resource that's already gone or unreachable.

Resolution.

kubectl get ns <org>-<app> -o json | jq '.status.conditions'
kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n1 kubectl get -n <org>-<app> --ignore-not-found

Find the resource still holding a finalizer and clear it once you've confirmed the upstream object is actually gone:

kubectl -n <org>-<app> patch <resource>/<name> \
  -p '{"metadata":{"finalizers":[]}}' --type=merge

Then re-run teardown from the platform so the report reconciles.

Prevention. Let the platform tear down in order — deprovision managed services and remove domains before deleting the app, so finalizers have their upstream dependencies removed first. The workflow does this when you delete top-down; manual kubectl delete ns skips the ordering.

4b. PVC not releasing

Symptom. Teardown completes for compute but the underlying disk/volume remains; a re-create later collides or storage cost lingers.

Root cause. The StorageClass reclaimPolicy is Retain, so deleting the PVC intentionally leaves the PV (and the cloud disk) behind.

Resolution.

kubectl get pv | grep <org>-<app>
kubectl get storageclass <sc> -o jsonpath='{.reclaimPolicy}'

If retention was intentional, archive/snapshot then delete the PV and the cloud disk manually. If it wasn't, set the class to Delete for future apps.

Prevention. Decide the reclaim policy per app deliberately — Retain for stateful data you can't lose, Delete for ephemeral workloads. The teardown report lists retained volumes explicitly so they're never a silent surprise; read it to completion.

4c. Confirming the teardown report

After a clean teardown the workflow emits a report of what it removed (namespace, registry repo, DNS records, managed services, identity bindings) and what it deliberately retained (e.g. Retain PVCs). Find it on the app's events / the workflow detail. A teardown isn't "done" until that report renders without an error step — if it's missing, teardown is still in flight or wedged on one of the cases above.

5. Secret rotation procedure

Rotating a credential without taking the app down.

Managed-service credential.

  1. On /apps/<slug>/managed-services, the active service owns its connection envelope (e.g. DATABASE_URL). Rotate the upstream credential through the service's rotation action; the platform updates the injected envelope and triggers a rolling restart so pods pick up the new value with zero-downtime overlap.
  2. Watch the resulting deployment in Deployments — the rollout keeps old pods serving until new pods (with the new credential) are healthy.

DeployToken (CI credential).

  1. On /apps/<slug>/tokens, click Rotate on the token. A new secret is shown once — copy it into your CI secret store.
  2. The previous secret stays valid for a 24-hour grace window (the default), so an in-flight CI run or a not-yet-updated pipeline doesn't break the moment you rotate.
  3. Update CI, then let the old secret lapse. Use Revoke (not Rotate) only when you need an immediate cutoff — Revoke breaks CI flows using that token right away.

Confirming rotation completed without downtime.

curl -I https://<app>.<base-domain>/health   # 200 throughout the rollout
  • Deployment shows succeeded, not a restart loop.
  • For DeployTokens, a CI run using the new secret authenticates; the old one still works until the grace window elapses, then 401s.

Prevention. Always Rotate for graceful handoff; reserve Revoke for compromise. Rotate CI tokens within the 24h window rather than letting them expire unattended. Use managed-service rotation actions rather than editing connection strings by hand, so the rolling restart is coordinated.

6. GKE credential expiry

Symptom. A registered GKE cluster flips to unreachable and its heartbeat stops. Health check returns an auth/token error. Deploys to that cluster queue or fail.

Root cause. The credential behind the cluster's source connection expired — a short-lived gcloud token wasn't refreshed, or the service account key was rotated/revoked upstream.

Resolution.

gcloud auth login                       # or: gcloud auth activate-service-account --key-file=key.json
gcloud container clusters get-credentials <cluster> \
  --region <region> --project <project>
kubectl get ns                          # confirm the refreshed creds work

Then update the stored credential on the cluster's SourceConnection (Settings → Source Providers, or the cluster registration) with the refreshed value, and re-run the cluster Health check. The heartbeat resumes once the new credential authenticates.

Prevention. Bind GKE access to Workload Identity (or a managed service account) rather than a user gcloud token, so the control plane holds a continuously-refreshable credential instead of one that expires out from under it.

7. EKS node group stuck scaling

Symptom. New pods sit Pending and the EKS managed node group won't scale out — replica count or new workloads can't be scheduled even though the autoscaler should be adding nodes.

Root cause. The Auto Scaling Group can't launch instances. The usual culprits: Spot capacity unavailable in the AZ/instance type, the AMI referenced by the launch template is missing/deregistered, or the node IAM role is missing required permissions (so nodes launch but never join, or never launch).

Resolution.

  1. Confirm pods are actually pending on scheduling, not on something else: 
    kubectl get pods -A --field-selector=status.phase=Pending
kubectl describe pod <pending-pod>   # "Insufficient ..." / "0/N nodes available"
  2. Check the ASG activity history for launch errors — AWS console → EC2 → Auto Scaling Groups → → Activity. Or: 
    aws autoscaling describe-scaling-activities \
  --auto-scaling-group-name <asg> --region <region> \
  --query 'Activities[?StatusCode!=`Successful`].[StatusCode,StatusMessage]'
aws eks describe-nodegroup --cluster-name <cluster> \
  --nodegroup-name <ng> --region <region> --query 'nodegroup.health'
  3. Map the error:
  4. Spot capacity (capacity-not-available / spot interruption): add fallback instance types to the node group, spread across more AZs, or add an on-demand node group as backstop.
  5. AMI not found: update the launch template / node group to a current EKS-optimized AMI release.
  6. IAM role missing: ensure the node role has the EKS worker, CNI, and ECR-read managed policies and a valid instance profile.

Prevention. Configure node groups with multiple instance types and AZs so a single Spot pool drying up doesn't block scheduling. Keep the cluster autoscaler / Karpenter healthy and its IAM permissions current. Pin a maintained AMI release channel rather than a specific AMI ID that can be deregistered. Watch the node group health field — EKS surfaces these exact causes there before pods start piling up.