Pipeline TOML DSL — Formal Spec (v1)¶

Status: Canonical · Issue #65 · Part of epic #64

All M1+ pipeline implementation tickets gate on this document. Changes to this spec require a new tracked issue.

1. File layout¶

<repo-root>/
  .astrolift/
    pipelines/
      ci.toml          # pipeline named "ci"
      nightly.toml     # pipeline named "nightly"
      deploy.toml      # pipeline named "deploy"

Directory: .astrolift/pipelines/
One TOML file per pipeline
Filename stem (without .toml) is the canonical pipeline name — ctx.pipeline.name
Multiple pipelines per repository are supported; each is an independent workflow

2. Top-level schema¶

A pipeline file has four top-level sections:

Section	Required	Description
`[pipeline]`	yes	Identity and metadata
`[on]`	yes	Trigger definitions
`[env]`	no	Pipeline-level environment variables
`[[jobs]]`	yes (≥ 1)	Ordered array of job definitions

3. `[pipeline]`¶

[pipeline]
name        = "ci"           # must match filename stem; alphanumeric + hyphens
description = "CI pipeline"  # optional, free text
version     = 1              # integer; only 1 is valid in v1

Field	Type	Required	Constraints
`name`	string	yes	`[a-z0-9-]+`, ≤ 64 chars, must equal filename stem
`description`	string	no	free text, ≤ 256 chars
`version`	integer	yes	must be `1`

Rationale: version is a forward-compatibility escape hatch. Parsers must reject files where version != 1 with a clear error rather than silently misinterpreting them.

4. `[on]` — Triggers¶

The [on] table is a collection of named sub-tables. At least one trigger is required. Multiple triggers may be specified; any matching trigger fires the pipeline.

4.1 Push trigger¶

[on.push]
branches = ["main", "release/*"]   # glob patterns; default: ["*"]
tags     = ["v*"]                  # glob patterns; default: []
paths    = ["src/**", "Dockerfile"] # optional path filter; default: all paths

Field	Type	Required	Default
`branches`	array of strings (glob)	no	`["*"]`
`tags`	array of strings (glob)	no	`[]`
`paths`	array of strings (glob)	no	all paths

A push event matches if: the ref matches at least one branches OR tags pattern AND (if paths is specified) at least one changed path matches.

4.2 Pull-request trigger¶

[on.pull_request]
branches     = ["main"]
events       = ["opened", "synchronize", "reopened"]
paths        = ["src/**"]

Field	Type	Required	Default
`branches`	array of strings (glob)	no	`["*"]`
`events`	array of strings	no	`["opened", "synchronize", "reopened"]`
`paths`	array of strings (glob)	no	all paths

Valid events values: opened, synchronize, reopened, closed, labeled, unlabeled.

4.3 Schedule trigger¶

[on.schedule]
cron = "0 2 * * *"   # UTC, standard 5-field cron

Field	Type	Required
`cron`	string	yes

Cron expressions use UTC. Five-field standard syntax only; no @reboot, @hourly aliases.

4.4 Manual trigger¶

[on.manual]
inputs = [
  { name = "environment", description = "Target environment", required = true, default = "staging" },
  { name = "dry_run",     description = "Skip deploy step",   required = false, default = "false" },
]

inputs is an optional array of input descriptors. Input values are available as ${env.<NAME>} at runtime (injected as environment variables before dispatch).

Input field	Type	Required
`name`	string	yes
`description`	string	no
`required`	boolean	no (default `false`)
`default`	string	no

5. `[env]` — Pipeline-level environment¶

[env]
NODE_ENV    = "production"
APP_VERSION = "${ctx.git.tag}"
BUILD_SHA   = "${ctx.git.sha}"

Key-value pairs injected into every job's environment. Values may contain ${...} substitution expressions (see section 8). Secrets must not appear as literal values here — use ${secrets.<NAME>} references instead.

6. `[[jobs]]` — Job definitions¶

[[jobs]] is a TOML array of tables. Each element is one job. Jobs are identified by their id field and may declare dependencies via needs.

6.1 Job identity and routing¶

[[jobs]]
id       = "test"
name     = "Run tests"        # optional display name; defaults to id
runs_on  = "astrolift/default"
needs    = []                 # upstream job ids this job waits on
timeout  = "30m"              # wall-clock timeout; default "1h"; max "24h"
continue_on_error = false     # if true, downstream jobs still run on failure

Field	Type	Required	Default
`id`	string	yes	—
`name`	string	no	value of `id`
`runs_on`	string	yes	—
`needs`	array of strings	no	`[]`
`timeout`	string (duration)	no	`"1h"`
`continue_on_error`	boolean	no	`false`

id constraints: [a-z0-9_-]+, unique within the pipeline, ≤ 64 chars.

Duration format: integer followed by s, m, or h — e.g. "90s", "30m", "2h".

6.2 `runs_on` taxonomy¶

runs_on controls where the job executes. The full value space:

Value	Meaning
`astrolift/default`	Scheduler picks any available cluster registered to the tenant's install
`astrolift/<region>`	Scheduler restricts to clusters in the given region slug (e.g. `astrolift/us-east-1`, `astrolift/eu-west-1`)
`cluster:<name>`	Route to a specific named cluster from the tenant's cluster registry (e.g. `cluster:prod-eks`)
`labels:{<key>:<value>, ...}`	Label-selector syntax; matches any cluster node pool whose labels satisfy all key-value pairs

Region slugs follow the cloud provider's region naming convention (AWS: us-east-1; GCP: us-central1; Azure: eastus). The install's cluster registry is the authoritative source — referencing an unknown region or cluster name is a dispatch-time error, not a parse-time error.

Label selector syntax:

runs_on = "labels:{tier:gpu,arch:amd64}"

Key-value pairs are comma-separated inside {}. No spaces around : or ,. Matching is exact (no regex or wildcard within a label value).

Routing resolution order:

cluster:<name> — direct, bypasses scheduler scoring
astrolift/<region> — scheduler picks the best cluster in that region
labels:{...} — scheduler picks the best cluster whose node pools match all labels
astrolift/default — scheduler picks globally

6.3 `[jobs.env]` — Job-level environment¶

[[jobs]]
id = "build"
runs_on = "astrolift/default"

[jobs.env]
DOCKER_BUILDKIT = "1"
IMAGE_TAG       = "${ctx.git.sha}"
DB_PASSWORD     = "${secrets.DB_PASSWORD}"

Job-level env is merged with pipeline-level env. Job-level values take precedence on collision. Secret references (${secrets.<NAME>}) are valid here; their resolved values are injected into the job's environment but are never echoed to logs.

6.4 `[[jobs.steps]]` — Step definitions¶

Steps are the atomic units of work within a job. They execute sequentially in declaration order.

[[jobs.steps]]
id               = "checkout"          # optional; required if other steps reference outputs
name             = "Checkout source"   # optional display name
run              = "git clone ..."     # shell command(s); multi-line via TOML triple-quote
shell            = "bash"             # default "bash"; also "sh", "python3"
working_dir      = "/workspace"        # default: job workspace root
env              = { EXTRA = "val" }   # step-level env overrides
if               = "${ctx.git.branch}" # step-level condition (non-empty string = run)
continue_on_error = false
timeout          = "10m"

Field	Type	Required	Default
`id`	string	no	—
`name`	string	no	—
`run`	string	yes	—
`shell`	string	no	`"bash"`
`working_dir`	string	no	job workspace root
`env`	inline table	no	`{}`
`if`	string	no	always run
`continue_on_error`	boolean	no	`false`
`timeout`	string (duration)	no	inherits job `timeout`

if condition semantics: The string value is evaluated after ${...} substitution. An empty string or the literal string "false" skips the step. Any other non-empty value runs the step. There is no boolean expression engine in v1.

run multi-line:

[[jobs.steps]]
name = "Build"
run  = """
set -euo pipefail
docker build -t myapp:${ctx.git.sha} .
docker push myapp:${ctx.git.sha}
"""

Step outputs:

A step may write key-value pairs to the file path in $ASTROLIFT_OUTPUTS_FILE (one KEY=VALUE per line). These values become available as ${jobs.<job-id>.outputs.<key>} in downstream jobs.

# Inside a step's run script
echo "image_tag=${IMAGE_TAG}" >> "$ASTROLIFT_OUTPUTS_FILE"

6.5 `[[jobs.services]]` — Service containers¶

Service containers run alongside the job's steps and are torn down when the job completes.

[[jobs.services]]
id    = "postgres"
image = "postgres:16"
env   = { POSTGRES_PASSWORD = "test", POSTGRES_DB = "testdb" }
ports = ["5432:5432"]

[jobs.services.health_check]
test     = ["pg_isready", "-U", "postgres"]
interval = "5s"
timeout  = "3s"
retries  = 10

Field	Type	Required	Default
`id`	string	yes	—
`image`	string	yes	—
`env`	inline table	no	`{}`
`ports`	array of strings	no	`[]`
`health_check`	table	no	no health check

Networking model: Service containers run as sidecars in the same pod as the job's step runner. They are reachable via localhost:<port>. There is no separate DNS name in v1.

Port mapping format: "<container-port>:<container-port>" — both sides must be the same value in v1. The notation exists for forward compatibility with host-port remapping.

Health-check semantics: Before the first step executes, the spawner waits until all service containers with a health_check pass (or until the job's timeout is reached, whichever comes first). test is an array: ["command", "arg1", "arg2"] executed inside the service container. The job fails immediately if any service fails its health check within the retry budget.

health_check fields:

Field	Type	Default
`test`	array of strings	— (required)
`interval`	duration string	`"10s"`
`timeout`	duration string	`"5s"`
`retries`	integer	`3`

6.6 `[jobs.outputs]` — Job output declarations¶

Outputs declared here are the interface of the job — the keys downstream jobs may reference via ${jobs.<id>.outputs.<key>}.

[jobs.outputs]
image_tag   = { description = "Docker image tag built by this job" }
release_sha = { description = "Git SHA of the released commit" }

Field	Type	Required
`<key>`	inline table	—
`<key>.description`	string	no

The values are populated at runtime from the step output file (see 6.4). Declaring an output key that no step populates resolves to an empty string (not an error). Referencing an undeclared output key from a downstream job is a parse-time error.

7. Dependency graph and execution model¶

needs forms a directed acyclic graph (DAG). Cycles are a parse-time error.
Jobs with no needs (or an empty needs array) are eligible to start as soon as the pipeline is dispatched.
A job becomes eligible when all jobs listed in its needs have completed successfully.
If a job fails and continue_on_error = false (the default), all downstream jobs that depend on it are skipped and marked as cancelled.
If a job fails and continue_on_error = true, downstream jobs still become eligible to run.
Jobs at the same eligibility level may execute in parallel, subject to tenant resource quota.

8. `${...}` substitution — expression language¶

v1 supports string interpolation only. Expressions are ${<path>} where <path> is a dot-separated reference into the context object graph.

8.1 Syntax rules¶

Delimiters: ${ and }
No nesting: ${ctx.git.${something}} is invalid
Expressions may appear inside any string-valued field
An unresolved reference (path not found in context) resolves to an empty string with a warning — it is not an error
A reference to ${secrets.<NAME>} resolves to the secret value in-process; the resolved value is never written to logs or substitution debug output

8.2 Context variable graph¶

The following variables are defined at each evaluation phase:

Phase 1 — Pipeline parse (on receive)¶

Available immediately when the pipeline file is loaded:

Variable	Type	Description
`ctx.git.sha`	string	Full 40-char commit SHA
`ctx.git.branch`	string	Branch name; empty string on a tag push
`ctx.git.tag`	string	Tag name; empty string when not a tag push
`ctx.git.repo`	string	`<org>/<repo>` slug
`ctx.git.actor`	string	Login of the user or service account that triggered the event
`ctx.pipeline.name`	string	Pipeline name (filename stem)
`ctx.pipeline.run_id`	string	Temporal workflow ID for this run
`ctx.pipeline.run_number`	integer	Monotonic run counter, per pipeline, per install
`env.<KEY>`	string	Pipeline-level env vars (literal values only at parse time)

Phase 2 — Job dispatch¶

All Phase 1 variables plus:

Variable	Type	Description
`ctx.job.id`	string	The `id` of the job being dispatched
`secrets.<NAME>`	string	Tenant secret value (injected, never logged)

Phase 3 — Step execution¶

All Phase 2 variables plus:

Variable	Type	Description
`jobs.<id>.outputs.<key>`	string	Output value from a completed upstream job

jobs.<id>.outputs.<key> is only valid for jobs listed in the current job's needs chain. Referencing an output from a job not in the dependency chain is a parse-time error.

8.3 Secret redaction rules¶

Secret values are resolved in-process by the job spawner; they are never written to the pipeline definition file, event payload, run logs, or substitution debug traces.
Log streams are post-processed by a redaction filter that replaces any occurrence of a known secret value with ***.
Substitution debug output (e.g. when ASTROLIFT_DEBUG_SUBSTITUTION=1) must omit the resolved values of any ${secrets.*} reference.

9. Artifact store contract¶

9.1 What an install must provide¶

Every Astrolift install that enables pipelines must provision a tenant-scoped blob store:

Cloud	Backend
AWS	S3 bucket, one per install
GCP	GCS bucket, one per install
Azure	Azure Blob Storage container, one per install
On-prem / K8s-native	MinIO instance or any S3-compatible endpoint

The install operator configures the blob store endpoint and credentials in the install manifest. The pipeline execution layer reads this configuration and handles upload/download transparently — pipeline authors do not manage credentials for artifact transfer.

9.2 Artifact lifecycle¶

Artifacts are keyed by <install-id>/<pipeline-name>/<run-id>/<job-id>/<artifact-name>
Retention: 30 days by default; configurable per install
Artifacts are private to the tenant — cross-tenant access is prohibited by IAM policy on the bucket

9.3 Step output file vs. artifact store¶

Two distinct mechanisms exist:

Mechanism	Use case	Size limit
Step output file (`$ASTROLIFT_OUTPUTS_FILE`)	Scalar key-value outputs (SHAs, version strings, flags)	64 KB
Artifact store (`$ASTROLIFT_ARTIFACTS_DIR`)	Binary files, build artifacts, test reports	Up to install quota (default 10 GB per run)

Artifact store usage from steps:

The execution layer mounts an empty directory at $ASTROLIFT_ARTIFACTS_DIR for the job. Files written there are uploaded to the blob store when the job completes. Downstream jobs receive $ASTROLIFT_ARTIFACTS_DIR pre-populated with the artifacts from all upstream jobs listed in needs.

# Upload: write to the artifacts dir during the job
cp dist/app.tar.gz "$ASTROLIFT_ARTIFACTS_DIR/app.tar.gz"

# Download: read from the artifacts dir in a downstream job
# (files from upstream jobs are already present)
tar -xzf "$ASTROLIFT_ARTIFACTS_DIR/app.tar.gz"

There is no explicit upload-artifact or download-artifact step action in v1. The execution layer handles this transparently based on the needs DAG.

10. Multi-tenant isolation contract¶

Pipeline workloads run on the same clusters as tenant application workloads. The spawner must enforce the following isolation contract. Implementation is M2's responsibility; this section defines what M2 must enforce.

10.1 Namespace isolation¶

Each pipeline run executes in a dedicated Kubernetes namespace: al-pipe-<run-id-short>
The namespace is created before the first job starts and deleted (with all resources) after the last job completes or the run times out
Namespace naming is deterministic and unique per run
Tenant application namespaces are separate; pipeline namespaces must not share a namespace with any app workload

10.2 Service account and RBAC¶

Each job pod runs under a dedicated service account scoped to its run namespace
The service account has no cluster-level permissions
Cross-namespace access is denied by default
Secret access within the namespace is limited to the secrets injected for that job

10.3 Resource quotas¶

Resource quotas are enforced at the namespace level. Default limits (overridable by install operator):

Resource	Default limit per pipeline run
CPU	8 cores
Memory	16 GiB
Ephemeral storage	50 GiB
Pods	20

Exceeding a quota causes the affected job to fail with a ResourceQuotaExceeded error; other jobs in the run are not affected.

10.4 Network policy¶

Pods in a pipeline run namespace may initiate egress to the public internet (subject to install-level egress policy)
Ingress to pipeline pods from outside the namespace is denied by default
Cross-namespace pod-to-pod traffic is denied by default
Service containers (sidecars) are reachable only from within the same pod via localhost

10.5 Node isolation (optional, operator-configured)¶

Install operators may dedicate node pools to pipeline workloads using Kubernetes node taints and tolerations. When a dedicated pipeline node pool is configured:

All pipeline job pods receive the toleration for the pipeline taint
Application workload pods do not receive the toleration
runs_on = "labels:{...}" can target the dedicated pool by label

This is optional. When no dedicated pool is configured, pipeline pods schedule on shared node pools subject to standard Kubernetes scheduling.

11. Secret injection model¶

11.1 Scope¶

By default, a pipeline job has access to all secrets in the tenant's secret store that are scoped to the install. No additional secrets: block is required in the TOML — all tenant secrets are available via ${secrets.<NAME>}.

Future versions may introduce an explicit secrets: allowlist per job. In v1, the implicit scope is full tenant secret access.

11.2 Injection mechanism¶

At job dispatch time, the spawner resolves ${secrets.<NAME>} references in the job's env table
Resolved secret values are injected as environment variables into the job pod via Kubernetes Secrets (not as ConfigMaps)
Kubernetes Secrets are created in the run namespace, bound to the job's service account, and deleted when the namespace is torn down
Secret values are never written to the pipeline definition, the run record, the Temporal workflow history, or any log stream

11.3 Redaction¶

The log ingestion pipeline applies a redaction pass before storing or streaming logs:

All known secret values for the run are collected at dispatch time
Any occurrence of a secret value in a log line is replaced with ***
Redaction applies to stdout, stderr, and any structured log fields

11.4 Secret reference resolution rules¶

Reference	Resolves to
`${secrets.MY_KEY}`	Value of secret `MY_KEY` in the tenant secret store
`${secrets.MISSING}`	Empty string with a warning; the job continues
A literal secret value in a `run` script	Not redacted at source; redacted in log output only

Operators should treat the log redaction pass as a safety net, not the primary protection. Pipeline authors must not construct secret values from string concatenation of non-secret parts.

12. Pydantic model reference¶

The following Pydantic v2 model is the normative schema definition. The TOML spec above and this model are kept in sync; on conflict, the Pydantic model takes precedence for implementation purposes.

from __future__ import annotations
from typing import Literal, Optional
from pydantic import BaseModel, Field, model_validator
import re

# ── Primitives ────────────────────────────────────────────────────────────────

DURATION_RE = re.compile(r"^\d+[smh]$")
ID_RE        = re.compile(r"^[a-z0-9_-]{1,64}$")

def _valid_duration(v: str) -> str:
    if not DURATION_RE.match(v):
        raise ValueError(f"Invalid duration: {v!r}. Expected form: 30m, 2h, 90s")
    return v

def _valid_id(v: str) -> str:
    if not ID_RE.match(v):
        raise ValueError(f"Invalid id: {v!r}. Must match [a-z0-9_-]{{1,64}}")
    return v

# ── Triggers ──────────────────────────────────────────────────────────────────

class PushTrigger(BaseModel):
    branches: list[str] = ["*"]
    tags:     list[str] = []
    paths:    list[str] = []

class PullRequestTrigger(BaseModel):
    branches: list[str] = ["*"]
    events:   list[str] = ["opened", "synchronize", "reopened"]
    paths:    list[str] = []

class ScheduleTrigger(BaseModel):
    cron: str

class ManualInput(BaseModel):
    name:        str
    description: str = ""
    required:    bool = False
    default:     str = ""

class ManualTrigger(BaseModel):
    inputs: list[ManualInput] = []

class OnBlock(BaseModel):
    push:         Optional[PushTrigger]         = None
    pull_request: Optional[PullRequestTrigger]  = None
    schedule:     Optional[ScheduleTrigger]     = None
    manual:       Optional[ManualTrigger]       = None

    @model_validator(mode="after")
    def at_least_one(self) -> "OnBlock":
        if all(v is None for v in [self.push, self.pull_request,
                                    self.schedule, self.manual]):
            raise ValueError("[on] must define at least one trigger")
        return self

# ── Service containers ────────────────────────────────────────────────────────

class HealthCheck(BaseModel):
    test:     list[str]
    interval: str = Field("10s", validate_default=True)
    timeout:  str = Field("5s",  validate_default=True)
    retries:  int = Field(3, ge=1, le=30)

    @model_validator(mode="after")
    def _durations(self) -> "HealthCheck":
        _valid_duration(self.interval)
        _valid_duration(self.timeout)
        return self

class Service(BaseModel):
    id:           str
    image:        str
    env:          dict[str, str] = {}
    ports:        list[str]      = []
    health_check: Optional[HealthCheck] = None

# ── Steps ─────────────────────────────────────────────────────────────────────

class Step(BaseModel):
    id:                 Optional[str]  = None
    name:               Optional[str]  = None
    run:                str
    shell:              str            = "bash"
    working_dir:        Optional[str]  = None
    env:                dict[str, str] = {}
    if_condition:       Optional[str]  = Field(None, alias="if")
    continue_on_error:  bool           = False
    timeout:            Optional[str]  = None

    @model_validator(mode="after")
    def _timeout_fmt(self) -> "Step":
        if self.timeout:
            _valid_duration(self.timeout)
        return self

    model_config = {"populate_by_name": True}

# ── Jobs ──────────────────────────────────────────────────────────────────────

class OutputDecl(BaseModel):
    description: str = ""

class Job(BaseModel):
    id:                str
    name:              Optional[str]          = None
    runs_on:           str
    needs:             list[str]              = []
    timeout:           str                    = Field("1h", validate_default=True)
    continue_on_error: bool                   = False
    env:               dict[str, str]         = {}
    services:          list[Service]          = []
    steps:             list[Step]
    outputs:           dict[str, OutputDecl]  = {}

    @model_validator(mode="after")
    def _validate(self) -> "Job":
        _valid_id(self.id)
        _valid_duration(self.timeout)
        return self

# ── Pipeline root ─────────────────────────────────────────────────────────────

class PipelineMeta(BaseModel):
    name:        str
    description: str    = ""
    version:     Literal[1]

    @model_validator(mode="after")
    def _name_fmt(self) -> "PipelineMeta":
        if not re.match(r"^[a-z0-9-]{1,64}$", self.name):
            raise ValueError("pipeline.name must match [a-z0-9-]{1,64}")
        return self

class Pipeline(BaseModel):
    pipeline: PipelineMeta
    on:       OnBlock
    env:      dict[str, str] = {}
    jobs:     list[Job]

    @model_validator(mode="after")
    def _dag(self) -> "Pipeline":
        ids = {j.id for j in self.jobs}
        for job in self.jobs:
            for dep in job.needs:
                if dep not in ids:
                    raise ValueError(
                        f"Job {job.id!r} needs unknown job {dep!r}"
                    )
        # cycle detection: Kahn's algorithm
        in_degree = {j.id: 0 for j in self.jobs}
        graph: dict[str, list[str]] = {j.id: [] for j in self.jobs}
        for job in self.jobs:
            for dep in job.needs:
                graph[dep].append(job.id)
                in_degree[job.id] += 1
        queue = [jid for jid, d in in_degree.items() if d == 0]
        visited = 0
        while queue:
            node = queue.pop()
            visited += 1
            for child in graph[node]:
                in_degree[child] -= 1
                if in_degree[child] == 0:
                    queue.append(child)
        if visited != len(self.jobs):
            raise ValueError("Cycle detected in job dependency graph")
        return self

    @model_validator(mode="after")
    def _output_refs(self) -> "Pipeline":
        declared: dict[str, set[str]] = {
            j.id: set(j.outputs.keys()) for j in self.jobs
        }
        needs_map: dict[str, list[str]] = {j.id: j.needs for j in self.jobs}
        import re as _re
        ref_pat = _re.compile(r"\$\{jobs\.([a-z0-9_-]+)\.outputs\.([a-z0-9_-]+)\}")
        for job in self.jobs:
            reachable = set(needs_map[job.id])
            for field_str in (str(job.env), str([s.env for s in job.steps])):
                for dep_id, key in ref_pat.findall(field_str):
                    if dep_id not in reachable:
                        raise ValueError(
                            f"Job {job.id!r} references output from "
                            f"{dep_id!r} which is not in its needs chain"
                        )
                    if key not in declared.get(dep_id, set()):
                        raise ValueError(
                            f"Job {job.id!r} references undeclared output "
                            f"{dep_id}.outputs.{key}"
                        )
        return self

13. Annotated example pipelines¶

13.1 Push-triggered test → build → deploy (3-job DAG)¶

[pipeline]
name    = "ci"
version = 1

[on.push]
branches = ["main"]

[env]
REGISTRY = "ghcr.io/myorg"

# ── Job 1: test ───────────────────────────────────────────────────────────────

[[jobs]]
id      = "test"
runs_on = "astrolift/default"
timeout = "20m"

[[jobs.services]]
id    = "postgres"
image = "postgres:16"
env   = { POSTGRES_PASSWORD = "test", POSTGRES_DB = "apptest" }
ports = ["5432:5432"]

[jobs.services.health_check]
test     = ["pg_isready", "-U", "postgres"]
interval = "5s"
timeout  = "3s"
retries  = 10

[[jobs.steps]]
name = "Install deps"
run  = "pip install -r requirements.txt"

[[jobs.steps]]
name = "Run tests"
run  = """
pytest --tb=short -q
"""
env  = { DATABASE_URL = "postgres://postgres:test@localhost:5432/apptest" }

# ── Job 2: build ──────────────────────────────────────────────────────────────

[[jobs]]
id      = "build"
needs   = ["test"]
runs_on = "astrolift/default"
timeout = "15m"

[jobs.env]
IMAGE = "${REGISTRY}/myapp:${ctx.git.sha}"

[jobs.outputs]
image_ref = { description = "Full image reference pushed to registry" }

[[jobs.steps]]
name = "Build and push image"
run  = """
set -euo pipefail
docker build -t "${IMAGE}" .
docker push "${IMAGE}"
echo "image_ref=${IMAGE}" >> "$ASTROLIFT_OUTPUTS_FILE"
"""

# ── Job 3: deploy ─────────────────────────────────────────────────────────────

[[jobs]]
id      = "deploy"
needs   = ["build"]
runs_on = "cluster:prod-eks"
timeout = "10m"

[jobs.env]
IMAGE_REF     = "${jobs.build.outputs.image_ref}"
KUBE_CONTEXT  = "prod"

[[jobs.steps]]
name = "Deploy to production"
run  = """
kubectl set image deployment/myapp app="${IMAGE_REF}" \
  --context "${KUBE_CONTEXT}" --namespace myapp-prod
kubectl rollout status deployment/myapp \
  --context "${KUBE_CONTEXT}" --namespace myapp-prod --timeout=5m
"""

13.2 PR trigger with status check¶

[pipeline]
name    = "pr-check"
version = 1

[on.pull_request]
branches = ["main"]
events   = ["opened", "synchronize"]

[[jobs]]
id      = "lint-and-test"
runs_on = "astrolift/default"
timeout = "15m"

[[jobs.steps]]
name = "Lint"
run  = "ruff check ."

[[jobs.steps]]
name = "Type check"
run  = "mypy src/"

[[jobs.steps]]
name = "Unit tests"
run  = "pytest tests/unit -q"

13.3 Scheduled nightly run¶

[pipeline]
name    = "nightly"
version = 1

[on.schedule]
cron = "0 2 * * *"   # 02:00 UTC daily

[env]
REPORT_BUCKET = "s3://myorg-reports"

[[jobs]]
id      = "integration-tests"
runs_on = "astrolift/default"
timeout = "2h"

[[jobs.steps]]
name = "Run full integration suite"
run  = """
set -euo pipefail
pytest tests/integration -v --junitxml=report.xml
cp report.xml "$ASTROLIFT_ARTIFACTS_DIR/report.xml"
"""

[[jobs]]
id      = "notify"
needs   = ["integration-tests"]
runs_on = "astrolift/default"
continue_on_error = true

[[jobs.steps]]
name = "Post results"
run  = """
curl -sS -X POST "${SLACK_WEBHOOK}" \
  -H "Content-Type: application/json" \
  -d "{\"text\": \"Nightly run ${ctx.pipeline.run_number} complete\"}"
"""
env  = { SLACK_WEBHOOK = "${secrets.SLACK_WEBHOOK_URL}" }

13.4 Job with service containers (postgres sidecar)¶

[pipeline]
name    = "db-migration-test"
version = 1

[on.push]
paths = ["migrations/**", "src/db/**"]

[[jobs]]
id      = "migration-test"
runs_on = "astrolift/default"
timeout = "10m"

[[jobs.services]]
id    = "postgres"
image = "postgres:16-alpine"
env   = { POSTGRES_USER = "app", POSTGRES_PASSWORD = "secret", POSTGRES_DB = "app" }
ports = ["5432:5432"]

[jobs.services.health_check]
test     = ["pg_isready", "-U", "app", "-d", "app"]
interval = "3s"
timeout  = "2s"
retries  = 15

[[jobs.services]]
id    = "redis"
image = "redis:7-alpine"
ports = ["6379:6379"]

[jobs.services.health_check]
test     = ["redis-cli", "ping"]
interval = "2s"
timeout  = "1s"
retries  = 10

[jobs.env]
DATABASE_URL = "postgres://app:secret@localhost:5432/app"
REDIS_URL    = "redis://localhost:6379/0"

[[jobs.steps]]
name = "Apply migrations"
run  = "alembic upgrade head"

[[jobs.steps]]
name = "Run migration tests"
run  = "pytest tests/test_migrations.py -v"

13.5 Artifact passing between jobs¶

[pipeline]
name    = "build-and-scan"
version = 1

[on.push]
branches = ["main", "release/*"]

# ── Job 1: compile ────────────────────────────────────────────────────────────

[[jobs]]
id      = "compile"
runs_on = "astrolift/default"
timeout = "20m"

[jobs.outputs]
binary_path = { description = "Path of compiled binary within artifacts dir" }

[[jobs.steps]]
name = "Build binary"
run  = """
set -euo pipefail
make build OUTPUT=dist/myapp
cp dist/myapp "$ASTROLIFT_ARTIFACTS_DIR/myapp"
echo "binary_path=myapp" >> "$ASTROLIFT_OUTPUTS_FILE"
"""

# ── Job 2: security scan ──────────────────────────────────────────────────────

[[jobs]]
id      = "security-scan"
needs   = ["compile"]
runs_on = "astrolift/default"
timeout = "10m"

[jobs.env]
BINARY = "${jobs.compile.outputs.binary_path}"

[[jobs.steps]]
name = "Scan binary"
run  = """
# $ASTROLIFT_ARTIFACTS_DIR is pre-populated with artifacts from "compile"
grype "$ASTROLIFT_ARTIFACTS_DIR/${BINARY}" --output json > scan-results.json
cp scan-results.json "$ASTROLIFT_ARTIFACTS_DIR/scan-results.json"
"""

[[jobs.steps]]
name = "Fail on critical CVEs"
run  = """
python3 -c "
import json, sys
with open('scan-results.json') as f:
    data = json.load(f)
criticals = [m for m in data.get('matches', [])
             if m.get('vulnerability', {}).get('severity') == 'Critical']
if criticals:
    print(f'{len(criticals)} critical CVE(s) found', file=sys.stderr)
    sys.exit(1)
"
"""

14. v1 non-goals¶

The following features are explicitly out of scope for v1. Do not spec, implement, or accept PRs for them until a new spec issue is opened and accepted.

Feature	Reason deferred
Caching (layer cache, dependency cache)	Requires cache key design, storage backend, and invalidation semantics; separate spec
Matrix strategy (`matrix:`)	Fan-out job graph; needs scheduler changes and a result-aggregation model
Composite / reusable actions (`uses:` pointing to user-defined repos)	Requires an action registry and version-pinning contract
Full expression evaluation engine	`${{ }}` style boolean/arithmetic expressions; security surface; deferred past M6
Conditional job execution (other than `continue_on_error`)	Depends on expression engine
Deployment environments with protection rules	Approval gates, deployment history; separate feature track
Concurrency groups (cancel-in-progress)	Queuing semantics not yet modelled
Workflow dispatch with typed inputs (beyond strings)	Typed inputs (boolean, choice, number) deferred with expression engine
Cross-pipeline triggers	Chaining pipelines via `workflow_call`-style triggers
Self-hosted runner registration	Bring-your-own compute registered outside Astrolift's cluster registry

15. Changelog¶

Date	Change
2026-05-30	Initial v1 spec (#65)