Skip to content
Docs

CI/CD Workflow

Overview

Hyve’s CI/CD mode separates the desired state (what you want) from the actual provisioning (making it happen). Engineers declare infrastructure changes locally; a pipeline running in your CI/CD system performs the cloud operations. This keeps credentials off developer machines and provides a full audit trail through Git history.

Local Mode

Default. hyve reconcile provisions clusters directly from the machine that runs the command. Good for local development and simple setups.

CI/CD Mode

hyve reconcile validates desired state and exits without provisioning — you push it, and a pipeline calls hyve reconcile --path . to do the actual provisioning. Credentials never leave the pipeline.

How It Works

Engineer declares a change

A team member adds or modifies a cluster definition and runs hyve reconcile locally.

Hyve validates and exits — you commit and push

In cicd mode, Hyve validates the YAML and exits without touching any cloud APIs. Hyve never commits or pushes on its own — see Version Control — so committing and pushing the desired state is your own explicit step: git commit -am "..." && git push.

Pipeline triggers

The push triggers a CI/CD workflow (e.g. GitHub Actions).

Pipeline reconciles

The pipeline checks out the repository and runs hyve reconcile --path .. Because --path is provided, Hyve always runs full local reconciliation regardless of the hyve.yaml mode setting.

Commit results back — your own step

Hyve writes reconciliation results (cluster status, driver outputs) back to local files. Committing and pushing them is a pipeline step you add explicitly (git add -A clusters cluster-state && git commit -m "hyve: reconcile" && git push), not something hyve does for you.

Repository Setup

Enable CI/CD Mode

Add a hyve.yaml file to the root of your Hyve state repository:

hyve.yaml
reconcile:
mode: cicd

This single file tells Hyve to skip local provisioning when hyve reconcile is called without --path.

Provider Scan (strictDelete)

By default, Hyve only ever acts on clusters that have a YAML definition in clusters/ — a cloud cluster it doesn’t know about is invisible to it, full stop.

Enable strictDelete to have Hyve additionally scan the cloud provider after each reconcile and warn about any cluster it finds that has no matching clusters/*.yaml:

hyve.yaml
reconcile:
mode: cicd
strictDelete: true
[strictDelete] Scanning 1 driver/region pair(s) for untracked clusters...
[strictDelete] Warning: cluster "leftover-demo" exists in ./custom-modules/civo@latest region=NYC1 but is not tracked in clusters/ — hyve will not touch it automatically; delete it manually if it shouldn't exist, or add a clusters/leftover-demo.yaml if it should be tracked
[strictDelete] Scan complete — 1 untracked cluster(s) found

This is deliberately warn-only — it never deletes anything. A cluster is too large a blast radius to remove automatically off a provider scan, especially since the scan itself can have blind spots: it only queries the driver module + region combinations already represented by some tracked cluster (there’s no standalone “every region this account has ever used” concept), so a cluster using a driver/region combination nothing else in clusters/ currently uses won’t be seen at all. If clusters/ is completely empty, there’s nothing to derive a scan scope from and the scan finds nothing.

To actually remove a cluster, either delete it explicitly (hyve cluster delete <name>, or spec.delete: true) or use scheduled expiry — see Deleting Clusters with onDelete Workflows below for why removing a cluster’s YAML file directly, without marking it deleted first, isn’t the recommended path either way.

hyve-state/ # Your Hyve state repository
├── hyve.yaml # Reconcile mode: cicd
├── hyve.lock # Locked module versions (commit this)
├── clusters/ # Cluster definitions
│ ├── production.yaml
│ ├── staging.yaml
│ └── development.yaml
├── templates/ # Reusable cluster templates
│ └── prod-template.yaml
└── workflows/ # Lifecycle workflow definitions
└── setup-monitoring.yaml

Example Cluster Definitions

# clusters/production.yaml — long-lived GKE cluster
apiVersion: hyve.io/v1alpha1
kind: ClusterDefinition
metadata:
name: production
spec:
region: us-central1
driver:
source: github.com/your-org/hyve-gke-module
version: v1.2.0
params:
project_id: my-gcp-project
machine_type: e2-standard-4
node_count: "3"
# clusters/dev-sprint-47.yaml — ephemeral cluster with automatic expiry
apiVersion: hyve.io/v1alpha1
kind: ClusterDefinition
metadata:
name: dev-sprint-47
spec:
region: PHX1
driver:
source: github.com/your-org/hyve-civo-module
version: v1.0.0
params:
node_size: g4s.kube.small
node_count: "1"
expiresAt: "2026-09-01T00:00:00Z" # auto-deleted after this time
workflows:
onDelete: [notify-team]
# clusters/staging.yaml — temporarily paused during migration
apiVersion: hyve.io/v1alpha1
kind: ClusterDefinition
metadata:
name: staging
spec:
region: us-east-1
driver:
source: github.com/your-org/hyve-eks-module
version: v2.1.3
params:
vpc_id: vpc-0abc123456789
eks_role_arn: arn:aws:iam::123456789012:role/eks-role
node_role_arn: arn:aws:iam::123456789012:role/node-role
instance_type: t3.medium
node_count: "3"
pause: true # reconciler skips this cluster until removed

GitHub Actions Workflows

Separate State Repository

Use this pattern when your Hyve state is kept in a dedicated repository, separate from your application repositories.

hyve-state/ # Dedicated state repository
├── .github/
│ └── workflows/
│ └── reconcile.yml
├── hyve.yaml # reconcile.mode: cicd
├── hyve.lock # Locked module versions
├── clusters/
│ ├── production.yaml
│ └── staging.yaml
├── templates/
│ └── prod-template.yaml
└── workflows/
└── setup-monitoring.yaml

.github/workflows/reconcile.yml:

name: Hyve Reconcile
on:
push:
branches: [main]
paths:
- 'clusters/**'
- 'templates/**'
- 'workflows/**'
- 'hyve.yaml'
- 'hyve.lock'
schedule:
- cron: '0 20 * * *' # Every day at 8:00 PM UTC
workflow_dispatch:
jobs:
reconcile:
runs-on: ubuntu-latest
permissions:
contents: write # Required to push reconciliation results back
steps:
- name: Checkout state repository
uses: actions/checkout@v4
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Configure git identity
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
- uses: actions/setup-go@v5
with:
go-version: "1.26"
- name: Install Hyve
run: |
go install github.com/cbridges1/hyve@latest
echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH"
- name: Install modules
run: hyve module install
- name: Reconcile clusters
env:
# Credentials are read by modules from standard env vars.
# Set whichever variables your modules expect.
# Civo
CIVO_TOKEN: ${{ secrets.CIVO_TOKEN }}
# AWS
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_REGION: ${{ secrets.AWS_REGION }}
# GCP
GOOGLE_APPLICATION_CREDENTIALS: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}
# Azure
AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}
AZURE_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }}
AZURE_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }}
run: hyve reconcile --path .
- name: Commit and push state
# hyve never commits or pushes — this records whatever reconcile
# changed: cluster-state/ sidecars, and the files of clusters it
# just deleted. if: always() keeps partial progress.
if: always()
run: |
git add -A clusters cluster-state
git diff --cached --quiet || { git commit -m "hyve: reconcile" && git push; }

Credentials are passed as standard environment variables. Modules read them directly — the same env vars used by the cloud CLIs. Add whichever secrets your modules require as GitHub Actions secrets.

Scheduled Reconcile

Add a schedule trigger directly to your existing reconcile workflow. This is the backbone for all time-based cluster automation — expiry-based deletion (expiresAt), normal drift correction, and paused-cluster awareness all happen through the same reconcile path.

on:
push:
branches: [main]
paths:
- 'clusters/**'
- 'hyve.yaml'
- 'hyve.lock'
schedule:
- cron: '0 */1 * * *' # Every hour — tightest window for expiresAt cluster cleanup
workflow_dispatch:

Adjust the cron expression to match your preferred cadence. Common schedules:

CronDescription
0 20 * * *Daily at 8:00 PM UTC
0 * * * *Hourly — tightest expiry window
0 0 * * *Daily at midnight UTC
0 8,20 * * *Twice daily at 8 AM and 8 PM UTC

Full Teardown

To destroy every cluster in the state repository on a schedule (e.g. overnight cost savings), mark each one delete: true before reconciling — the same explicit, per-cluster mechanism Deleting Clusters with onDelete Workflows below covers, just applied in bulk. This is not what strictDelete is for: it only warns about clusters with no YAML at all, so removing every clusters/*.yaml file and relying on it to clean up the cloud side would do nothing but print warnings.

.github/workflows/nightly-teardown.yml
name: Nightly Teardown
on:
schedule:
- cron: '0 0 * * *' # Midnight UTC
workflow_dispatch:
jobs:
teardown:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Configure git identity
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
- uses: actions/setup-go@v5
with:
go-version: "1.26"
- name: Install Hyve
run: |
go install github.com/cbridges1/hyve@latest
echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH"
- name: Mark every cluster for deletion
# spec.delete: true, not a deleted file — the reconciler needs the
# YAML to still be there to know which onDelete workflows to run and
# against which driver/params, exactly as in the mark-for-deletion
# pattern below. It removes the file locally once teardown
# completes; the commit step at the end pushes that removal.
run: |
for f in clusters/*.yaml clusters/*.yml; do
[ -f "$f" ] || continue
yq -i '.spec.delete = true' "$f"
done
git add clusters/
git commit -m "Nightly teardown: mark all clusters for deletion" || echo "Nothing to commit"
git push
- name: Reconcile (deletes every cluster marked above)
env:
PROD_US_AWS_ACCESS_KEY_ID: ${{ secrets.PROD_US_AWS_ACCESS_KEY_ID }}
PROD_US_AWS_SECRET_ACCESS_KEY: ${{ secrets.PROD_US_AWS_SECRET_ACCESS_KEY }}
MY_PROJECT_GCP_CREDENTIALS_JSON: ${{ secrets.MY_PROJECT_GCP_CREDENTIALS_JSON }}
MY_ORG_CIVO_TOKEN: ${{ secrets.MY_ORG_CIVO_TOKEN }}
run: hyve reconcile --path .
- name: Commit and push state
# hyve never commits or pushes — this records whatever reconcile
# changed: cluster-state/ sidecars, and the files of clusters it
# just deleted. if: always() keeps partial progress.
if: always()
run: |
git add -A clusters cluster-state
git diff --cached --quiet || { git commit -m "hyve: reconcile" && git push; }

Deleting Clusters with onDelete Workflows

Deleting the YAML file for a cluster directly means the reconciler never sees the cluster definition — and therefore cannot run any onDelete workflows before the cluster is removed. To safely delete a cluster and have its lifecycle workflows execute, use the mark-for-deletion pattern instead.

How It Works

Set spec.delete: true in the cluster YAML without removing the file:

clusters/staging.yaml
apiVersion: hyve.io/v1alpha1
kind: ClusterDefinition
metadata:
name: staging
spec:
delete: true # <-- marks this cluster for deletion
driver:
source: github.com/your-org/hyve-eks-module
version: v2.1.3
region: us-east-1
params:
instance_type: t3.medium
node_count: "3"
workflows:
onDelete:
- cleanup-namespaces
- drain-nodes

When the reconciler processes this file it will:

Detect the deletion marker

DetermineAction returns ActionDelete as soon as it sees spec.delete: true — no cloud API call is needed to make this decision.

Run onDelete workflows

All workflows listed under spec.workflows.onDelete are executed against the cluster before any cloud resources are touched.

Delete the cluster

The cluster is removed from the cloud provider.

Delete the cluster

The cluster is removed from the cloud provider.

Remove the YAML file and push

The reconciler removes the cluster definition file from the repository, commits the change, and pushes it back to the remote. The deletion marker is never left behind.

Required GitHub Actions Configuration

The reconciler removes a deleted cluster’s file (and its cluster-state/ sidecar) locally; your pipeline’s commit step pushes that back. So the workflow needs write access to the repository and a git committer identity.

1. Contents write permission

jobs:
reconcile:
runs-on: ubuntu-latest
permissions:
contents: write

2. Git committer identity

GitHub Actions runners do not configure a git identity by default. Add this step before the commit step:

- name: Configure git identity
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"

3. Branch protection rules

If your default branch requires pull request reviews or passing checks before merging, the pipeline’s direct push will be rejected.

Choose one of:

  • Exempt the Actions bot — add github-actions[bot] to the bypass list in your branch protection settings.

  • Use a Personal Access Token (PAT) — pass it to the checkout step so the push is made as a user with bypass permissions.

    - uses: actions/checkout@v4
    with:
    token: ${{ secrets.HYVE_PAT }}

Complete example

name: Hyve Reconcile
on:
push:
branches: [main]
paths:
- 'clusters/**'
- 'templates/**'
- 'workflows/**'
schedule:
- cron: '0 20 * * *' # Every day at 8:00 PM UTC
workflow_dispatch:
jobs:
reconcile:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Checkout state repository
uses: actions/checkout@v4
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Configure git identity
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
- uses: actions/setup-go@v5
with:
go-version: "1.26"
- name: Install Hyve
run: |
go install github.com/cbridges1/hyve@latest
echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH"
- name: Reconcile clusters
env:
PROD_US_AWS_ACCESS_KEY_ID: ${{ secrets.PROD_US_AWS_ACCESS_KEY_ID }}
PROD_US_AWS_SECRET_ACCESS_KEY: ${{ secrets.PROD_US_AWS_SECRET_ACCESS_KEY }}
run: hyve reconcile --path .
- name: Commit and push state
# hyve never commits or pushes — this records whatever reconcile
# changed: cluster-state/ sidecars, and the files of clusters it
# just deleted. if: always() keeps partial progress.
if: always()
run: |
git add -A clusters cluster-state
git diff --cached --quiet || { git commit -m "hyve: reconcile" && git push; }

Pausing Reconciliation

Set spec.pause: true to keep a cluster’s definition in the repository while telling the reconciler to skip it entirely. The cluster continues running in the cloud — Hyve simply does not compare desired state or make any changes until pause is removed.

clusters/staging.yaml
apiVersion: hyve.io/v1alpha1
kind: ClusterDefinition
metadata:
name: staging
spec:
pause: true # reconciler skips this cluster on every run
driver:
source: github.com/your-org/hyve-civo-module
version: v1.0.0
region: NYC1
params:
node_size: g4s.kube.medium

Common use cases:

  • Maintenance windows — prevent Hyve from touching a cluster while you perform manual operations
  • Cost investigation — keep the definition versioned while you decide whether to delete or resize
  • Temporary divergence — allow a cluster to drift from its YAML definition without triggering an update

To resume normal reconciliation, remove the pause: true line (or set it to false) and commit.


Scheduled Cluster Expiry

Set spec.expiresAt to an RFC 3339 timestamp to have the reconciler automatically delete a cluster when that time passes. When the scheduled reconcile runs after the expiry time, the reconciler treats the cluster exactly as if spec.delete: true had been set — running onDelete workflows, deleting from the cloud provider, and removing the YAML file.

clusters/dev-sprint-47.yaml
apiVersion: hyve.io/v1alpha1
kind: ClusterDefinition
metadata:
name: dev-sprint-47
spec:
driver:
source: github.com/your-org/hyve-civo-module
version: v1.0.0
region: PHX1
params:
node_size: g4s.kube.small
expiresAt: "2026-05-01T00:00:00Z" # deleted automatically after this time
workflows:
onDelete:
- notify-team
- cleanup-namespaces

The expiry is checked on every reconcile run. With an hourly scheduled reconcile, clusters are cleaned up within ~1 hour of their deadline.

Combining pause and expiresAt:

A paused cluster is always skipped — the expiry check is never reached. If you pause a cluster that has an expiresAt set, it will not be deleted until pause is removed.

Scheduled Reconcile Workflow

No special pipeline logic is needed. The schedule trigger on your existing reconcile workflow handles drift correction, expirations, and deletions in a single run — no separate workflow file required.

The expiry check runs on every reconcile. With a daily schedule at 8 PM, expired clusters are cleaned up within 24 hours of their deadline. Use an hourly schedule (0 * * * *) if you need tighter expiry windows.

Ephemeral Environment Pattern

A common pattern for feature-branch or sprint clusters:

clusters/feature-login-redesign.yaml
apiVersion: hyve.io/v1alpha1
kind: ClusterDefinition
metadata:
name: feature-login-redesign
spec:
driver:
source: github.com/your-org/hyve-eks-module
version: v2.1.3
region: us-east-1
params:
instance_type: t3.medium
node_count: "2"
expiresAt: "2026-04-25T18:00:00Z" # end of sprint
workflows:
onCreate:
- deploy-feature-branch
onDelete:
- export-test-results
- notify-slack

When the sprint ends, the cluster tears itself down automatically — no manual cleanup required.


End-to-End Workflow

Here is the complete flow from developer change to provisioned cluster:

Terminal window
# 1. Developer: create a new cluster definition locally, from a template
hyve cluster create staging --template gcp-standard \
--region us-central1 --set node_count=2
# 2. Developer: run reconcile (in cicd mode — validates, does NOT provision)
hyve reconcile
# Output:
# Reconcile mode: cicd
# Skipping local reconciliation — cluster provisioning will be handled by the CI/CD pipeline.
# 💡 Push the desired state yourself so the CI/CD pipeline picks it up: git add -A && git commit -m ... && git push
# 3. Developer: push it
git add clusters/staging.yaml && git commit -m "Add staging" && git push
# 4. GitHub Actions triggers on push to main and runs: hyve reconcile --path .
# Cluster 'staging' is created in GCP (us-central1)
# 5. The pipeline's commit step pushes cluster-state/staging.state.yaml back

Credential Configuration

Modules read credentials from environment variables — the same env vars used by their underlying cloud CLIs. Set them as GitHub Actions secrets.

Standard credential environment variables

ProviderEnvironment variables
CivoCIVO_TOKEN
AWSAWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION
GCPGOOGLE_APPLICATION_CREDENTIALS (path to service account JSON), or use OIDC
AzureAZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET

You can also pass credentials as module params via cluster.spec.params, but environment variables are preferred for CI/CD since they are never written to Git.

Required GitHub Secrets

Set these in your repository’s Settings → Secrets and variables → Actions.

SecretDescription
GITHUB_TOKENAutomatically provided by GitHub Actions; needs contents: write permission
Provider credentialsAdd whichever credential env vars your modules expect (see table above)

Security Considerations

Use OIDC for AWS and GCP

Instead of storing long-lived AWS/GCP credentials as secrets, use GitHub Actions’ OIDC provider with AWS IAM roles or GCP Workload Identity Federation. This eliminates the need to rotate secrets.

permissions:
id-token: write
contents: read
Limit repository permissions

Grant the contents: write permission only to the reconcile workflow, not globally. Use branch protection rules to require PR reviews before merging cluster changes to main.

Audit trail through Git history

Because all state changes flow through Git, every cluster change has an associated commit, author, and timestamp. Use git log clusters/ to see the full history.

Review changes before they apply

Use pull requests for cluster changes. Engineers open a PR modifying clusters/*.yaml; reviewers approve; the merge to main triggers the pipeline. This gives you a review gate before anything is provisioned.

Troubleshooting

Pipeline fails: Civo API token not found

Hyve has no built-in provider-credentials mechanism — each module reads whatever environment variables its own driver expects (for the Civo module, typically CIVO_TOKEN). Make sure the CI job exports that variable from a secret before running hyve:

# workflow env:
env:
CIVO_TOKEN: ${{ secrets.MY_ORG_CIVO_TOKEN }}
Pipeline fails: permission denied pushing to repository

Ensure the workflow has contents: write permission:

permissions:
contents: write

Or use a token with write access in the checkout step.

Local hyve reconcile provisions instead of pushing

Check that hyve.yaml is committed to the root of your state repository with reconcile.mode: cicd. If the file is absent, Hyve defaults to local mode.

onDelete workflows did not run before cluster was deleted

This happens when the cluster YAML file was deleted directly from the repository instead of using the spec.delete: true marker. The reconciler never saw the file, so it had no workflow list to execute — the cluster appeared as an unmanaged orphan and was removed without running lifecycle hooks.

Prevention: Always set spec.delete: true in the YAML file and let the reconciler handle the deletion. Do not git rm or manually delete cluster files when onDelete workflows need to run.

Pipeline fails: permission denied pushing cluster file removal

The pipeline’s commit step needs write access to push the cluster file removal after spec.delete: true is processed.

Ensure the workflow has contents: write permission and a git identity is configured:

permissions:
contents: write
steps:
- name: Configure git identity
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"

If branch protection rules block direct pushes, see Branch protection rules.

hyve reconcile --path . fails: no clusters found

Verify that cluster YAML files exist under a clusters/ directory at the path you provided. The --path flag should point to the root of the repository, not the clusters/ subdirectory.

Cluster with expiresAt was not deleted on schedule

The expiry check runs on every reconcile. Verify:

  1. The scheduled reconcile workflow is running — check the Actions tab in GitHub.
  2. The expiresAt value is a valid RFC 3339 timestamp: "2026-05-01T00:00:00Z". An invalid format logs a warning and skips the expiry check.
  3. The cluster does not also have pause: true set — a paused cluster is skipped entirely before the expiry check is reached.
  4. The workflow has contents: write, a git identity, and a commit step, so the YAML file removal after deletion is pushed back.
Paused cluster is being modified by the reconciler

If a cluster with pause: true is still being reconciled, verify that the change has been committed and pushed to the branch the pipeline checks out. The reconciler reads the YAML files from the checked-out repository — a local change that has not been pushed will not take effect in the pipeline.