Release smoke scenarios¶

These scenarios are a practical release checklist for maintainers and a source of small, repeatable examples for evaluators. They deliberately avoid invented output and hard-coded credentials. Record the release, server URL, organization, source ref, image digest, and observed IDs with each run.

Use a disposable organization, project, repositories, and secret values. Do not run destructive scenarios against a customer workload.

1. Local control-plane baseline¶

From the astrolift-app repository:

./bootstrap.sh
./run.sh up
./run.sh seed
./run.sh urls
./run.sh ps

Expected checks:

the frontend loads at http://localhost:3000;
the backend health endpoint is http://localhost:8000/health/;
the GraphQL endpoint loads at http://localhost:8000/app/gql/config/;
Temporal UI loads at http://localhost:8233; and
the seeded development identity can open the dashboard.

Run the backend test suite inside the Compose stack:

./run.sh test

If the schema changed, also verify the checked-in schema and frontend generated types using the commands documented in astrolift-app/bootstrap.md.

2. CLI discovery and authentication¶

Use the exact CLI binary intended for the release:

astro version
astro --help
astro server --help
astro auth --help
astro app --help
astro agent --help
astro workflow --help
astro pipeline --help

Add and select the target server, then complete device authentication:

astro server add test https://astrolift.example.com
astro server use test
astro auth login

Confirm an authenticated read succeeds. Then deliberately run one write with a read-only token and verify that it fails. API-token scopes are a ceiling over RBAC grants: an administrator using a read-only device token must not gain write access.

When automating, pass the organization explicitly and prefer JSON output:

astro --org <org-slug> --json app list

Do not put tokens or secret values in the shell history captured with the test record.

3. Register and deploy an app¶

In a test repository containing a valid astrolift.toml:

astro app register \
  --project-id <project-guid> \
  --source-repo <owner/repo> \
  --manifest-path astrolift.toml

astro --app <app-slug> app deploy \
  --image-tag <immutable-image-tag-or-digest> \
  --env production \
  --wait

Validate all five app concerns in the dashboard:

Build: source connection, manifest status, and deployment history;
Run: workload state, replicas, routes, terminal, and commands;
Observe: logs, metrics, events, and alerts;
Control: environment, scaling, rollback, and lifecycle actions; and
Secure: secrets, bundles, identity, policy, and audit information.

Pushing a repository does not imply a deployment by itself. Confirm whether the source webhook and managed CI workflow are installed, which branch is watched, and whether the workflow only builds or also calls the deployment API. Capture the GitHub/GitLab run and the corresponding Astrolift deployment ID.

For cancellation behavior, start two CI builds in quick succession and verify the repository workflow's concurrency policy cancels the older build. This is a source-workflow behavior unless an implemented Astrolift pipeline concurrency policy is explicitly configured.

4. Secret CRUD and bundles¶

In the app's Secure area:

Create a disposable secret.
Verify list/status never returns plaintext.
Reveal the value only if the selected backing provider advertises value reveal; otherwise the UI must explain that reveal is unavailable.
Update/rotate the value and verify history/audit metadata changes.
Add the secret to a bundle and attach the bundle at the intended scope.
Remove the binding and delete the disposable value.

Repeat one write with the CLI or API. The token must carry secret.write; the user's organization role alone is insufficient. A read operation needs the matching read scope and still respects provider reveal limitations.

Never use a real signing key, production database password, or customer secret for this scenario.

5. Register an agent repository¶

An agent repository can contain one root manifest or several manifests in a monorepo. Register explicit paths first so the test has a bounded blast radius:

astro agent register-repo <owner/repo> \
  --project-id <project-guid> \
  --ref <immutable-ref> \
  --manifest-path agents/triage/astrolift.toml

Confirm that:

only the selected manifest subtree is reconciled;
the registered workload has kind agent and task-family execution;
referenced prompts, scripts, binaries, and local skills cannot escape the allowed source tree;
repository and catalogue skills resolve to the intended versions; and
secret references are present while secret values are absent from the manifest snapshot and Brief.

For a monorepo test, register a second manifest independently. Update one path and confirm the other agent is not unintentionally modified or deployed.

6. Create an environment recipe and secret packet¶

Create or update an environment recipe. An explicit image wins over a catalogue runtime, so choose only one for a simple test:

astro agent env-spec upsert triage-test \
  --agent-type claude \
  --image-tag <agent-image-digest> \
  --config-repo <owner/repo> \
  --config-branch <ref> \
  --manifest-path agents/triage/astrolift.toml \
  --tool-preset dev \
  --secret JIRA_TOKEN=triage-test-jira

Set the value without a command-line argument:

printf '%s' "$TRIAGE_TEST_JIRA_TOKEN" | \
  astro agent secret set triage-test JIRA_TOKEN --stdin

astro agent secret ls triage-test

The status command should show set/missing state, never the value. Verify the write fails with a token that lacks secret.write, even if that token belongs to an administrator.

If the environment uses a cloud-managed model, enable that option through the supported UI/API and verify the task pod receives workload identity rather than a long-lived model API key. Invoke the configured cloud model before pinning it; a model catalogue record can remain visible after the provider marks the model legacy.

7. Dispatch, observe, and cancel an agent¶

Start a bounded task and stream its logs:

astro agent dispatch <agent-slug> \
  --env-spec triage-test \
  --input '{"mode":"smoke","limit":1}' \
  --timeout 600 \
  --tail

Verify:

the task progresses through queued, provisioning, running, and a terminal state;
the pod has an active deadline matching the task timeout;
application stdout/stderr appears in astro agent logs, not only Xvnc or a terminal supervisor;
an interactive terminal or VNC session is a separate stream and still works;
AGENT_CALLBACK_URL and a task-scoped callback token are injected;
findings/interactions and the terminal result reach the task record; and
the per-task Kubernetes Secret is removed after completion.

Then dispatch a deliberately long-running test task, find its ID, and cancel it:

astro agent ls
astro agent inspect <task-id>
astro agent cancel <task-id> --yes

For a RUNNING or PROVISIONING Kubernetes task, confirm the Job and pod disappear before the task becomes CANCELLED. This is the release gate for runaway-agent control.

8. Compose a chained agent workflow¶

Scaffold a file or use this minimal worked example:

[workflow]
slug = "triage-chain"
name = "Triage Chain"
pattern = "chained"
description = "Collect evidence, then classify it."

[[stage]]
kind = "agent_dispatch"
role = "investigator"
agent = "evidence-agent"
environment_spec_slug = "triage-test"
skills = ["collect-evidence"]
prompt = "Collect reproducible evidence for the supplied report."
output_key = "evidence"
timeout = 600

[[stage]]
kind = "agent_dispatch"
role = "triager"
agent = "classification-agent"
skills = ["classify-finding"]
prompt = "Classify and deduplicate the evidence from the prior stage."
output_key = "classification"
timeout = 600

agent resolves an organization-local registered agent workload. environment_spec_slug selects that stage's environment recipe. skills adds skill references to the stage Brief. prompt is an immutable instruction overlay for that dispatch. output_key must be unique and names the result in the workflow's accumulated output map; a chained stage also receives the prior stage result directly. model_label is intentionally absent because it is a legacy Django content-object target, not an LLM or agent model.

Validate locally and on the target server before import:

astro workflow validate workflow.toml
astro workflow validate workflow.toml --server
astro workflow import workflow.toml --preview
astro workflow import workflow.toml

An imported definition is a template. Create a configured workflow when stage bindings or default inputs are required, then run it:

astro workflow create \
  --definition triage-chain \
  --name "Triage chain smoke" \
  --trigger manual

astro workflow run triage-chain-smoke --input issue=TEST-1
astro workflow runs triage-chain-smoke --watch

If a stage's agent slug already resolves, explicit --bind flags are not needed. If the definition is role-only or you want a different concrete agent, bind each agent stage with --bind <stage-order>=<agent-workload-guid>.

9. Webhooks¶

Test source-provider and outbound webhooks separately:

a source webhook is inbound, provider-signed, and drives source sync, build, pipeline, or workflow routing;
an outbound app webhook is emitted by Astrolift and signed with the subscription secret described in Webhook signature verification.

Outbound application webhooks are not currently a runnable demo gate. The control plane's stored-key signing defect means a receiver configured with the returned plaintext secret cannot verify the dashboard's signed test action, and the asynchronous delivery path is not wired to an HTTP sender. See the linked verification guide for the intended contract and the exact limitation. After the defect is fixed, verify that a receiver rejects a changed body, a stale timestamp, and a wrong secret, then verify rotation only accepts the previous secret during the configured overlap.

For a repository source webhook, deliver a real provider test event and verify its delivery ID is deduplicated. Replaying the same delivery must not create a second deployment or agent dispatch.

10. Pipeline reality check¶

Use astro pipeline list, run, runs, logs, and cancel only against a persisted pipeline definition. The toml_path metadata and experimental fetch helpers do not yet make repository pipeline TOML an end-to-end public contract. See Pipeline definitions before demonstrating this feature.

The pipeline smoke gate is:

a manual run creates one PipelineRun;
jobs respect their persisted needs DAG;
independent jobs run in parallel;
an eligible self-hosted runner claims at most one job;
logs and outputs remain organization-scoped; and
cancellation reaches the Temporal workflow and active runner.

11. Documentation and SDK validation¶

From the docs repository:

mkdocs build --strict

From the CLI repository:

go test ./...

From the control-plane repository, run provider protocol and parity tests as part of ./run.sh test. A provider plugin passing discovery alone is not enough: the protocol signatures, declared capabilities, managed-service availability matrix, and real credential smoke tests must agree.

Release evidence¶

For each scenario, retain:

Astrolift release and CLI versions;
immutable application and agent image digests;
organization/project/app/agent slugs;
source repository and commit SHA;
deployment, task, workflow-run, or pipeline-run IDs;
terminal state and relevant audit-event IDs; and
redacted logs that demonstrate the expected boundary.

Never attach plaintext secret values, callback tokens, deploy tokens, provider credentials, or customer payloads to the release record.