Pipeline definitions: current implementation status¶

The pipeline TOML is read and executed, but is not a public contract yet

A pipeline run does now fetch, parse, and persist the TOML. On every run, mark_pipeline_run_running reads the document at the pipeline's toml_path, parses it with the one canonical reader, reconciles Job and Step rows from it, and records the document's SHA-256 digest and declared schema version on the run.

What is still missing before this can be documented as a contract you build automation against: the CLI does not validate with the server's rules, sync does not pin an immutable source ref, and matrices, expressions and outputs have no end-to-end execution tests. See Requirements for a stable native DSL for the current state of each.

Until those close, prefer the dashboard/API-managed objects for anything you cannot afford to have change under you.

This page records the behavior that exists today and the boundary around the unfinished declarative interface.

Pipeline execution model¶

A pipeline belongs to one organization and may optionally be associated with a registered app. Its stored definition consists of:

Object	Current fields
`Pipeline`	name, repository URL, default branch, TOML path, optional registered app
`Job`	job ID, display name, runner selector, container image, dependency job IDs
`Step`	position, optional step ID, exactly one action reference or shell command, environment, action parameters
`Trigger`	push, pull request, schedule, manual, or webhook plus kind-specific JSON configuration
`PipelineRun`	run number, trigger source, source ref/actor, Temporal workflow ID, status and timestamps
`JobRun` / `StepRun`	execution status, assigned runner, output, logs, timing, and failure data

Jobs form a directed acyclic graph through needs. The Temporal PipelineRunWorkflow runs a topological tier in parallel and waits before starting dependent jobs. Circular or missing dependencies are invalid.

A step can either:

invoke a registered built-in action through uses; or
execute a shell command through run.

Those alternatives are mutually exclusive at the definition boundary, even though the database permits partial rows while a definition is being built.

Runners¶

Self-hosted runners poll the controller API for matching work. A runner declares its operating system, architecture, and arbitrary labels. runs_on is matched against those labels. Registration uses a one-time token; successful registration consumes it and returns the long-lived runner credential. Only an IDLE runner can claim a job.

Runner endpoints are organization-scoped and validate the runner credential on every claim, heartbeat, log, and completion call. Do not reuse an Astrolift user or deploy token as a runner credential.

Runs, logs, and cancellation¶

The supported CLI surface is discoverable from the installed binary:

astro pipeline --help
astro pipeline list
astro pipeline run <pipeline>
astro pipeline runs --pipeline <pipeline>
astro pipeline logs <run-id>
astro pipeline cancel <run-id>

Run identifiers and flags can change between CLI releases; use the command's --help before scripting it. --json is the stable choice for machine-readable output.

Cancellation marks the run cancelled and signals its Temporal workflow. The workflow then stops scheduling remaining work and attempts to cancel active jobs. A runner may take a short time to observe cancellation, so callers should wait for terminal job states instead of assuming the CLI return means every child process has already exited.

Triggers and source authentication¶

Pipeline triggers are persisted objects. Push and pull-request triggers are routed from source-provider webhooks, schedules are synchronized to the durable scheduler, and manual runs can be started from the dashboard, API, or CLI.

GitHub and GitLab webhook paths validate provider signatures and delivery metadata before creating a run. Generic outbound application webhooks are a different feature; their signing contract is documented in Webhook signature verification.

Trigger filters belong to the trigger's configuration, not an undocumented TOML table. Inspect the API schema or dashboard generated by the installed release for the accepted JSON shape.

Secrets and expressions¶

Pipeline secrets are resolved for the organization and optional app at dispatch time and passed only to the selected job runner. Secret values must not be put in job environment records, logs, trigger JSON, or repository metadata.

The execution layer supports context expansion for selected values, including source information and outputs from completed jobs. An output reference fails closed when the job or key does not exist; it is not silently replaced with an empty string. Treat values derived from untrusted source events as untrusted input to shell commands.

Because no full native TOML parser is wired into pipeline startup, helper code such as dsl_extensions.py is not evidence that every proposed expression, conditional, matrix, output, or schema-version field is accepted end to end.

Concurrency behavior¶

The codebase contains a concurrency-policy hook and organization run-rate limits. The per-organization rate limit is active where the trigger path invokes it. The default is 60 newly created runs per minute unless the organization overrides it.

The proposed TOML block below is not active configuration today:

[concurrency]
group = "deploy-production"
cancel_in_progress = true

get_concurrency_config() currently returns no group and cancel_in_progress = false; it does not parse the pipeline's repository file. If you need “new build cancels old build” now, enforce it in the source provider's workflow (for example GitHub Actions concurrency) or through an operator-owned pipeline definition that has an implemented concurrency field.

The repository TOML path¶

Pipeline.toml_path defaults to:

.astrolift/pipelines/<pipeline-name>.toml

GitHub and GitLab fetch helpers can retrieve that path for a run ref, but no production call site currently invokes the fetcher. The field is therefore metadata and a reservation for the declarative sync feature, not proof that editing or pushing the file auto-deploys a pipeline.

Similarly, the converter can render GitHub Actions or GitLab CI input into an intermediate TOML representation. Conversion output is not automatically a valid, deployable native pipeline definition. Treat conversion as preview or migration assistance until server-side validation and import are wired to the same schema.

Requirements for a stable native DSL¶

The declarative pipeline feature should not be considered supported until all of these are true in one release. Current state:

#	Requirement	State
1	One published schema defines the accepted document and version field	Done. `schema_version` is declared on every written document and validated on read. A document with no field is read as version 1, so files written before the field existed keep working. An unrecognised version is refused with the accepted set in the message.
2	The CLI validates with the same rules as the server	Not done. `astro workflow validate --server` does this for workflow TOML; there is no pipeline equivalent.
3	Registration/sync fetches a file at an immutable source ref	Not done. The fetch uses the run's trigger ref, which is a branch and therefore mutable.
4	Parsing reconciles `Pipeline`, `Job`, `Step` and `Trigger` rows and reports structured path/line errors	Mostly. `Job` and `Step` are reconciled per run, and every parse failure carries the offending key path. `Trigger` rows are not reconciled from the document.
5	A pipeline run records the source ref and definition digest it executed	Done. `PipelineRun.definition_digest` (SHA-256 of the document text) and `definition_schema_version`, both stamped only after the definition persists, so a run never claims a definition that failed to load. Empty and `0` mean "no document behind this run", which is a real state for an API-defined pipeline.
6	Round-trip and backward-compatibility tests cover every documented field	Partly. Round-trip is now byte-idempotent (write → read → write is identical) and compatibility of unversioned documents is pinned. Per-field coverage is not complete.
7	Repository webhooks state whether a push only syncs configuration or also starts a run	Not done.
8	Concurrency, matrices, expressions, outputs, secrets, and action inputs have end-to-end execution tests rather than isolated helper tests	Not done. Secrets have an end-to-end path; the rest do not.

Composition¶

A pipeline document is a single file, and a repository holds one per pipeline at .astrolift/pipelines/<name>.toml. There is no include, extends, or import: the top-level keys are schema_version, name, on, env and jobs, and the only reuse primitive is a step's uses reference to a registered action.

Splitting a large definition across files is therefore a directory convention, not a language feature — the same shape workflow TOML already uses, where every workflows/**/*.toml file is discovered in deterministic order and parsed independently. Adding true composition would need a documented merge order, cycle detection, and an immutable ref for any remote include, and would make requirement 5 mandatory rather than useful: with N files, "what did this run execute" has no answer without the digest.

Until then, use the dashboard/API-managed objects and the current astro pipeline run-management commands. Use astro workflow for the separate, implemented workflow-definition TOML contract used to compose agent and human stages.