Pipeline definitions: current implementation status¶

The pipeline TOML is read and executed, but is not a public contract yet

A pipeline run does now fetch, parse, and persist the TOML. On every run, mark_pipeline_run_running reads the document at the pipeline's toml_path, parses it with the one canonical reader, reconciles Job and Step rows from it, and records the document's SHA-256 digest and declared schema version on the run.

What is still missing before this can be documented as a contract you build automation against: the CLI does not validate with the server's rules, a manual run without a commit fetches a moving branch, triggers are not reconciled from the document, and matrices, expressions and outputs have no end-to-end execution tests. See Requirements for a stable native DSL for the current state of each.

Until those close, prefer the dashboard/API-managed objects for anything you cannot afford to have change under you.

This page records the behavior that exists today and the boundary around the unfinished declarative interface. There is no astro pipeline validate; the reader is backend/astrolift_ci_convert/common/toml_reader.py in astrolift-app, and a run's recorded definition_digest is the record of what it read.

Pipeline execution model¶

A pipeline belongs to one organization and may optionally be associated with a registered app. Its stored definition consists of:

Object	Current fields
`Pipeline`	name, repository URL, default branch, TOML path, optional registered app
`Job`	job ID, display name, runner selector, container image, dependency job IDs
`Step`	position, optional step ID, exactly one action reference or shell command, environment, action parameters
`Trigger`	push, pull request, schedule, manual, or webhook plus kind-specific JSON configuration
`PipelineRun`	run number, trigger source, source ref/actor, Temporal workflow ID, status and timestamps
`JobRun` / `StepRun`	execution status, assigned runner, output, logs, timing, and failure data

Jobs form a directed acyclic graph through needs. The Temporal PipelineRunWorkflow runs a topological tier in parallel and waits before starting dependent jobs. Circular or missing dependencies are invalid.

A step can either:

invoke a registered built-in action through uses; or
execute a shell command through run.

Those alternatives are mutually exclusive at the definition boundary, even though the database permits partial rows while a definition is being built.

Runners¶

A job's runs_on selects where it runs. The default, astrolift/default, and any selector naming astrolift or an OS/architecture pair run as an unprivileged Kubernetes Job on one of the organization's clusters whose node OS, architectures and labels satisfy it; with no matching cluster the job fails to schedule and says so. Because a cluster job has no Docker daemon and cannot escalate, uses = "astrolift/docker-build" is refused there at render time; build on a self-hosted runner or in your own CI.

self-hosted and macos select a self-hosted runner and are never spawned on a cluster. Self-hosted runners poll the controller API for matching work. A runner declares its operating system, architecture, and arbitrary labels, and runs_on is matched against those. Registration authenticates with the installation's runner registration token, configured by the operator; with no token configured, registration is disabled rather than open. Registration returns the long-lived runner credential. Only an idle runner can claim a job.

Runner endpoints are organization-scoped and validate the runner credential on every claim, heartbeat, log, and completion call. Do not reuse an Astrolift user or deploy token as a runner credential.

Runs, logs, and cancellation¶

The supported CLI surface is discoverable from the installed binary:

astro pipeline --help
astro pipeline list
astro pipeline run <pipeline> --request-file start.json --yes
astro pipeline runs --pipeline <pipeline>
astro pipeline show <run-guid>
astro pipeline logs <run-id>
astro pipeline cancel <run-id> --yes
astro pipeline reconcile --request-file start.json

run is a reviewed start: it binds the run to the pipeline's current version and keeps a private request file so an interrupted start can be read back with reconcile instead of resubmitted. See Reviewed starts. Run identifiers and flags can change between CLI releases; use the command's --help before scripting it. --json is the stable choice for machine-readable output.

Cancellation marks the run cancelled and signals its Temporal workflow. The workflow then stops scheduling remaining work and attempts to cancel active jobs. A runner may take a short time to observe cancellation, so callers should wait for terminal job states instead of assuming the CLI return means every child process has already exited.

Triggers and source authentication¶

Pipeline triggers are persisted objects. Push and pull-request triggers are routed from source-provider webhooks, schedules are synchronized to the durable scheduler, and manual runs can be started from the dashboard, API, or CLI.

GitHub and GitLab webhook paths validate provider signatures and delivery metadata before creating a run. Generic outbound application webhooks are a different feature; their signing contract is documented in Webhook signature verification.

Trigger filters belong to the trigger's configuration, not an undocumented TOML table. Inspect the API schema or dashboard generated by the installed release for the accepted JSON shape.

Secrets and expressions¶

Pipeline secrets are resolved for the organization and optional app at dispatch time and reach only the job that references them, as a Kubernetes Secret that lives as long as the job pod. A run from a fork gets none. Secret values must not be put in job environment records, logs, trigger JSON, or repository metadata.

The execution layer supports context expansion for selected values, including source information and outputs from completed jobs. An output reference fails closed when the job or key does not exist; it is not silently replaced with an empty string. Treat values derived from untrusted source events as untrusted input to shell commands.

The reader accepts env and outputs tables, but no production path evaluates ${...} expressions, if conditionals or job outputs when a job is spawned; helper code for those exists without a caller. Do not build on them.

Concurrency behavior¶

The codebase contains a concurrency-policy hook and two rate limits. The one that is enforced is the webhook receiver's: 300 deliveries per organization per minute, checked after the signature, and a delivery over it is answered 429. The 60-runs-per-minute run limit and the concurrency hook have no caller on any trigger path.

The proposed TOML block below is not active configuration today:

[concurrency]
group = "deploy-production"
cancel_in_progress = true

get_concurrency_config() currently returns no group and cancel_in_progress = false; it does not parse the pipeline's repository file, and nothing calls the policy it feeds. If you need "new build cancels old build" now, enforce it in the source provider's workflow (for example GitHub Actions concurrency).

The repository TOML path¶

Pipeline.toml_path defaults to:

.astrolift/pipelines/<pipeline-name>.toml

When a run starts, the document is fetched at the run's commit SHA, falling back to the trigger ref and then the pipeline's default branch when no commit was recorded (a manual run started from a branch). GitHub and GitLab repositories are read through the organization's source connection; any other Git host fails the run with a clear error. A fetch or parse failure fails the run rather than reporting an empty one as green.

Editing or pushing the file does not create or change the Pipeline or its Trigger rows; those are made through the dashboard, API or CLI, and the document is read only when a run of that pipeline starts.

Similarly, the converter can render GitHub Actions or GitLab CI input into an intermediate TOML representation. Conversion output is not automatically a valid, deployable native pipeline definition. Treat conversion as preview or migration assistance until server-side validation and import are wired to the same schema.

Requirements for a stable native DSL¶

The declarative pipeline feature should not be considered supported until all of these are true in one release. Current state:

#	Requirement	State
1	One published schema defines the accepted document and version field	Done. `schema_version` is declared on every written document and validated on read. A document with no field is read as version 1, so files written before the field existed keep working. An unrecognised version is refused with the accepted set in the message.
2	The CLI validates with the same rules as the server	Not done. `astro workflow validate --server` does this for workflow TOML; there is no pipeline equivalent.
3	Registration/sync fetches a file at an immutable source ref	Mostly. A webhook-started run fetches at its commit SHA. A manual run started from a branch records no commit and fetches the branch, which moves.
4	Parsing reconciles `Pipeline`, `Job`, `Step` and `Trigger` rows and reports structured path/line errors	Mostly. `Job` and `Step` are reconciled per run, and every parse failure carries the offending key path. `Trigger` rows are not reconciled from the document.
5	A pipeline run records the source ref and definition digest it executed	Done. `PipelineRun.definition_digest` (SHA-256 of the document text) and `definition_schema_version`, both stamped only after the definition persists, so a run never claims a definition that failed to load. Empty and `0` mean "no document behind this run", which is a real state for an API-defined pipeline.
6	Round-trip and backward-compatibility tests cover every documented field	Partly. Round-trip is now byte-idempotent (write → read → write is identical) and compatibility of unversioned documents is pinned. Per-field coverage is not complete.
7	Repository webhooks state whether a push only syncs configuration or also starts a run	Not done.
8	Concurrency, matrices, expressions, outputs, secrets, and action inputs have end-to-end execution tests rather than isolated helper tests	Not done. Secrets have an end-to-end path; the rest do not.

Document shape as read¶

The reader accepts these keys and refuses anything it cannot type, naming the offending path. on is read into the parsed definition but is not reconciled into Trigger rows (requirement 4), and env and outputs are stored without being evaluated (see Secrets and expressions).

schema_version = 1
name = "orders-api"

[on.push]
branches = ["main"]
tags = ["v*"]

[on.pull_request]
branches = ["main"]

[[on.schedule]]
cron = "0 3 * * *"

[on.workflow_dispatch]

[env]
CI = "true"

[jobs.test]
name = "Test"
runs_on = "astrolift/default"
container = "python:3.12"
needs = []

  [[jobs.test.steps]]
  uses = "astrolift/git-checkout@v1"

  [[jobs.test.steps]]
  name = "Unit tests"
  run = "make test"

    [jobs.test.steps.env]
    PYTHONDONTWRITEBYTECODE = "1"

Key	Meaning
`schema_version`	Integer; `1` is the only accepted value and the default when absent
`name`	Required
`on.push.branches`, `on.push.tags`, `on.pull_request.branches`	String arrays
`on.schedule`	Array of tables, each with a `cron`
`on.workflow_dispatch`	Presence enables manual starts
`env`	String map; an unquoted number is read as its string
`jobs.<id>`	`name` (defaults to the id), `runs_on` (default `astrolift/default`), `container`, `needs`, `env`, `outputs`, `steps`
`jobs.<id>.steps[]`	`name`, exactly one of `uses` or `run`, `env`, `with` (action parameters)

Jobs are read in sorted id order; needs carries the real ordering. The built-in actions are astrolift/git-checkout@v1, astrolift/docker-build@v1, astrolift/kubectl-apply@v1 and astrolift/astrolift-deploy@v1; the unversioned name resolves to @v1. An unknown action, or a missing required input, fails the job before its pod is created.

Composition¶

A pipeline document is a single file, and a repository holds one per pipeline at .astrolift/pipelines/<name>.toml. There is no include, extends, or import: the top-level keys are schema_version, name, on, env and jobs, and the only reuse primitive is a step's uses reference to a registered action.

Splitting a large definition across files is therefore a directory convention, not a language feature, the same shape workflow TOML already uses, where every workflows/**/*.toml file is discovered in deterministic order and parsed independently. Adding true composition would need a documented merge order, cycle detection, and an immutable ref for any remote include, and would make requirement 5 mandatory rather than useful: with N files, "what did this run execute" has no answer without the digest.

Until then, use the dashboard/API-managed objects and the current astro pipeline run-management commands. Use astro workflow for the separate, implemented workflow-definition TOML contract used to compose agent and human stages.