Release smoke scenarios¶

These scenarios are a practical release checklist for maintainers and a source of small, repeatable examples for evaluators. They deliberately avoid invented output and hard-coded credentials. Record the release, server URL, organization, source ref, image digest, and observed IDs with each run.

Use a disposable organization, project, repositories, and secret values. Do not run destructive scenarios against a customer workload.

Placeholders used below: <org-slug> is shown by astro org current; <project-guid> is the id of a project in astro project list --json; <app-slug> is the slug printed by astro app register; <task-id> is the AgentTask ID printed by astro agent dispatch.

1. Local control-plane baseline¶

From the astrolift-app repository:

./bootstrap.sh
./run.sh up
./run.sh seed
./run.sh urls
./run.sh ps

Expected checks:

the frontend loads at http://localhost:3000;
the backend health endpoint is http://localhost:8000/health/;
the GraphQL endpoint loads at http://localhost:8000/app/gql/config/;
Temporal UI loads at http://localhost:8233; and
the seeded development identity can open the dashboard.

Run the backend test suite inside the Compose stack:

./run.sh test

If the schema changed, also verify the checked-in schema and frontend generated types using the commands documented in astrolift-app/bootstrap.md.

2. CLI discovery and authentication¶

Use the exact CLI binary intended for the release:

astro version
astro --help
astro server --help
astro auth --help
astro app --help
astro agent --help
astro workflow --help
astro pipeline --help

Add and select the target server, then complete device authentication:

astro server add test https://astrolift.example.com
astro server use test
astro auth login
astro org list
astro org use <org-slug>
astro whoami

astro server add prints Added server test -> https://astrolift.example.com. astro auth login ends with Logged in to https://astrolift.example.com.; astro auth status reports Not logged in. when it did not. astro whoami prints Server, Org, User and Email lines for the credential in use.

Confirm an authenticated read succeeds. Then deliberately run one write with a read-only token and verify that it fails with api token scope does not allow this permission. API-token scopes are a ceiling over RBAC grants: an administrator using a read-only token must not gain write access.

When automating, pass the organization explicitly and prefer JSON output:

astro --org <org-slug> --json app list

Do not put tokens or secret values in the shell history captured with the test record.

3. Register and deploy an app¶

In a test repository containing a valid astrolift.toml:

astro app register \
  --project-id <project-guid> \
  --source-repo <owner/repo> \
  --manifest-path astrolift.toml \
  --yes

astro --app <app-slug> app deploy \
  --image-tag <immutable-image-tag-or-digest> \
  --env production \
  --wait

astro app register prints Registered app: <name> (<slug>); without --yes it first asks Use this? [Y/n/edit] about the detected build context. No managed cluster connected. means the organization has no managed tenant cluster; register one before continuing. astro app deploy --wait prints Deployment started: <id>, each status transition, and Final status: running; failed, superseded or rolled_back exits 1.

Validate the app's tabs in the dashboard:

Overview: status and topology;
Deployments: history, approval state, and previews;
Workloads: workload state, replicas, scheduled jobs, and managed services;
Logs & metrics: logs, metrics, console, and one-shot commands;
Domains: hostname bindings and validation state;
Secrets: literal secrets and attached bundles;
Access: members, deploy tokens, security scans, and edge access; and
Settings: configuration, manifest, webhooks, environments, and the danger zone.

Pushing a repository does not imply a deployment by itself. Confirm whether the source webhook and managed CI workflow are installed, which branch is watched, and whether the workflow only builds or also calls the deployment API. Capture the GitHub/GitLab run and the corresponding Astrolift deployment ID.

For cancellation behavior, start two CI builds in quick succession and verify the repository workflow's concurrency policy cancels the older build. This is a source-workflow behavior unless an implemented Astrolift pipeline concurrency policy is explicitly configured.

4. Secret CRUD and bundles¶

In the app's Secrets tab:

Create a disposable secret.
Verify list/status never returns plaintext.
Reveal the value only if the selected backing provider advertises value reveal; otherwise the UI must explain that reveal is unavailable.
Update/rotate the value and verify history/audit metadata changes.
Add the secret to a bundle and attach the bundle at the intended scope.
Remove the binding and delete the disposable value.

Repeat one write with the CLI or API:

printf '%s' "$SMOKE_VALUE" | astro app secrets create SMOKE_KEY <app-slug> --stdin
astro app secrets list <app-slug>
astro app secrets delete SMOKE_KEY <app-slug> --yes

The write prints Created secret SMOKE_KEY on app <app-slug>.; the list shows the key with its environment, scope and SET VIA, never the value. The token must carry the secret:write scope and the caller the secret.write permission; the user's organization role alone is insufficient, and the refusal reads api token scope does not allow this permission. A read operation needs the matching read scope and still respects provider reveal limitations.

Never use a real signing key, production database password, or customer secret for this scenario.

5. Register an agent repository¶

An agent repository can contain one root manifest or several manifests in a monorepo. Register explicit paths first so the test has a bounded blast radius:

astro agent register-repo <owner/repo> \
  --project-id <project-guid> \
  --ref <immutable-ref> \
  --manifest-path agents/triage/astrolift.toml

Confirm that:

only the selected manifest subtree is reconciled;
the registered workload has kind agent and spawns as a Kubernetes Job on dispatch rather than rolling through the deploy pipeline;
referenced prompts, scripts, binaries, and local skills cannot escape the allowed source tree;
repository and catalogue skills resolve to the intended versions; and
secret references are present while secret values are absent from the manifest snapshot and Brief.

For a monorepo test, register a second manifest independently. Update one path and confirm the other agent is not unintentionally modified or deployed.

6. Create an environment recipe and secret packet¶

Create or update an environment recipe. An explicit image wins over a catalogue runtime, so choose only one for a simple test:

astro agent env-spec upsert triage-test \
  --agent-type claude \
  --image-tag <agent-image-digest> \
  --config-repo <owner/repo> \
  --config-branch <ref> \
  --manifest-path agents/triage/astrolift.toml \
  --tool-preset dev \
  --secret JIRA_TOKEN=triage-test-jira

Set the value without a command-line argument:

printf '%s' "$TRIAGE_TEST_JIRA_TOKEN" | \
  astro agent secret set triage-test JIRA_TOKEN --stdin

astro agent secret ls triage-test

The status command should show set/missing state, never the value. Verify the write fails with a token that lacks the secret:write scope, even if that token belongs to an administrator.

If the environment uses a cloud-managed model, enable it on the agent's Configuration › Model access section and verify the task pod receives workload identity rather than a long-lived model API key. Invoke the configured cloud model before pinning it; a model catalogue record can remain visible after the provider marks the model legacy.

7. Dispatch, observe, and cancel an agent¶

Start a bounded task and stream its logs:

astro agent dispatch <agent-slug> \
  --env-spec triage-test \
  --input '{"mode":"smoke","limit":1}' \
  --timeout 600 \
  --tail

The command prints AgentTask ID: <task-id> and Status: <status>, then each transition until Final status: <status>; a failed run exits 1.

Verify:

the task progresses through queued, provisioning, running, and a terminal state;
the pod's activeDeadlineSeconds matches the task timeout;
application stdout/stderr appears in astro agent logs <task-id>, not only Xvnc or a terminal supervisor;
an interactive terminal or VNC session is a separate stream and still works (astro agent vnc, astro agent session attach <task-id>);
AGENT_CALLBACK_URL, the task-scoped ASTROLIFT_CLUSTER_KEY, and the ASTROLIFT_MESSAGES_URL and ASTROLIFT_RUN_MCP_URL endpoints are injected;
a question the agent asks reaches astro inbox list --unread and the dashboard inbox (/inbox);
findings/interactions and the terminal result reach the task record; and
the per-task Kubernetes Secret is removed after completion.

Then dispatch a deliberately long-running test task, find its ID, and cancel it:

astro agent ls --status running
astro agent inspect <task-id>
astro agent cancel <task-id> --yes

For a RUNNING or PROVISIONING Kubernetes task, confirm the Job and pod disappear before the task becomes CANCELLED. This is the release gate for runaway-agent control.

8. Compose a chained agent workflow¶

Scaffold a file with astro workflow init or use this minimal worked example:

[workflow]
slug = "triage-chain"
name = "Triage Chain"
pattern = "chained"
description = "Collect evidence, then classify it."

[[stage]]
kind = "agent_dispatch"
role = "investigator"
agent = "evidence-agent"
environment_spec_slug = "triage-test"
skills = ["collect-evidence"]
prompt = "Collect reproducible evidence for the supplied report."
output_key = "evidence"
timeout = 600

[[stage]]
kind = "agent_dispatch"
role = "triager"
agent = "classification-agent"
skills = ["classify-finding"]
prompt = "Classify and deduplicate the evidence from the prior stage."
output_key = "classification"
timeout = 600

agent resolves an organization-local registered agent workload. environment_spec_slug selects that stage's environment recipe. skills adds skill references to the stage Brief. prompt is an immutable instruction overlay for that dispatch. output_key must be unique and names the result in the workflow's accumulated output map; a chained stage also receives the prior stage result directly. model_label is intentionally absent because it is a legacy Django content-object target, not an LLM or agent model. The Workflow TOML reference lists every key.

Validate locally and on the target server before import:

astro workflow validate workflow.toml
astro workflow validate workflow.toml --server
astro workflow import workflow.toml --preview
astro workflow import workflow.toml

Local validation prints OK (local shape check): workflow.toml; --server prints OK (server validation): and the parsed structure. --preview prints Preview OK and persists nothing. The persisting import prints Imported workflow definition: <slug>; re-importing without --replace creates <slug>-1, <slug>-2, and so on, while --replace upserts the organization's own definition.

An imported definition is a template. Create a configured workflow when stage bindings or default inputs are required, then run it:

astro workflow create \
  --definition triage-chain \
  --name "Triage chain smoke" \
  --trigger manual

astro workflow run triage-chain-smoke --input issue=TEST-1
astro workflow runs triage-chain-smoke --watch

create prints Created workflow: Triage chain smoke (triage-chain-smoke); run prints a Run ID: line; runs --watch polls every 5 seconds and reports the newest run's state until it is terminal.

If a stage's agent slug already resolves, explicit --bind flags are not needed. If the definition is role-only or you want a different concrete agent, bind each agent stage with --bind <stage-order>=<agent-workload-guid>.

9. Webhooks¶

Test source-provider and outbound webhooks separately:

a source webhook is inbound, provider-signed, and drives source sync, build, pipeline, or workflow routing;
an outbound app webhook is emitted by Astrolift and signed with the subscription secret described in Webhook signature verification.

Outbound application webhooks are not currently a runnable demo gate. The control plane persists only the SHA-256 digest of the subscription secret and uses that digest as the HMAC key, so a receiver configured with the returned plaintext secret cannot verify the dashboard's signed test action or the asynchronous deliveries the HTTP sender emits. See the linked verification guide for the intended contract and the exact limitation. After the defect is fixed, verify that a receiver rejects a changed body, a stale timestamp, and a wrong secret, then verify rotation only accepts the previous secret during the configured overlap.

For a repository source webhook, deliver a real provider test event and verify its delivery ID is deduplicated. Replaying the same delivery must not create a second deployment or agent dispatch.

10. Pipeline reality check¶

Use astro pipeline list, run, runs, show, logs, and cancel only against a persisted pipeline definition. astro pipeline run requires --request-file <private-path> and --yes; keep the request file, because astro pipeline reconcile reads the original start from it without resubmitting. The toml_path metadata and fetch helpers do not yet make repository pipeline TOML an end-to-end public contract. See Pipeline definitions before demonstrating this feature.

The pipeline smoke gate is:

a manual run creates one PipelineRun;
jobs respect their persisted needs DAG;
independent jobs run in parallel;
an eligible self-hosted runner claims at most one job;
logs and outputs remain organization-scoped; and
cancellation reaches the Temporal workflow and active runner.

11. Documentation and SDK validation¶

From the docs repository:

mkdocs build --strict
python3 scripts/check_local_links.py
python3 scripts/check_toml_examples.py

From the CLI repository:

go test ./...

From the control-plane repository, run provider protocol and parity tests as part of ./run.sh test. A provider plugin passing discovery alone is not enough: the protocol signatures, declared capabilities, managed-service availability matrix, and real credential smoke tests must agree.

Release evidence¶

For each scenario, retain:

Astrolift release and CLI versions;
immutable application and agent image digests;
organization/project/app/agent slugs;
source repository and commit SHA;
deployment, task, workflow-run, or pipeline-run IDs;
terminal state and relevant audit-event IDs; and
redacted logs that demonstrate the expected boundary.

Never attach plaintext secret values, callback tokens, deploy tokens, provider credentials, or customer payloads to the release record.