CI/CD Workflow

Overview

Hyve’s CI/CD mode separates the desired state (what you want) from the actual provisioning (making it happen). Engineers declare infrastructure changes locally; a pipeline running in your CI/CD system performs the cloud operations. This keeps credentials off developer machines and provides a full audit trail through Git history.

Local Mode

Default. hyve reconcile provisions clusters directly from the machine that runs the command. Good for local development and simple setups.

CI/CD Mode

hyve reconcile validates and pushes desired state to Git, then exits. A pipeline calls hyve reconcile --path . to do the actual provisioning. Credentials never leave the pipeline.

How It Works

Engineer declares a change

A team member adds or modifies a cluster definition and runs hyve reconcile locally.

Hyve validates and pushes state

In cicd mode, Hyve validates the YAML, commits the desired state to the Git repository, and exits without touching any cloud APIs.

Pipeline triggers

The push to the state repository triggers a CI/CD workflow (e.g. GitHub Actions).

Pipeline reconciles

The pipeline checks out the state repository and runs hyve reconcile --path .. Because --path is provided, Hyve always runs full local reconciliation regardless of the hyve.yaml mode setting.

Changes committed back

Hyve writes reconciliation results (cluster status, kubeconfigs, etc.) back to the repository and pushes them.

Repository Setup

Enable CI/CD Mode

Add a hyve.yaml file to the root of your Hyve state repository:

hyve.yaml

reconcile:
  mode: cicd

This single file tells Hyve to skip local provisioning when hyve reconcile is called without --path.

Provider Scan (strictDelete)

By default, Hyve only ever acts on clusters that have a YAML definition in clusters/ — a cloud cluster it doesn’t know about is invisible to it, full stop.

Enable strictDelete to have Hyve additionally scan the cloud provider after each reconcile and warn about any cluster it finds that has no matching clusters/*.yaml:

hyve.yaml

reconcile:
  mode: cicd
  strictDelete: true

[strictDelete] Scanning 1 driver/region pair(s) for untracked clusters...
[strictDelete] Warning: cluster "leftover-demo" exists in ./custom-modules/civo@latest region=NYC1 but is not tracked in clusters/ — hyve will not touch it automatically; delete it manually if it shouldn't exist, or add a clusters/leftover-demo.yaml if it should be tracked
[strictDelete] Scan complete — 1 untracked cluster(s) found

This is deliberately warn-only — it never deletes anything. A cluster is too large a blast radius to remove automatically off a provider scan, especially since the scan itself can have blind spots: it only queries the driver module + region combinations already represented by some tracked cluster (there’s no standalone “every region this account has ever used” concept), so a cluster using a driver/region combination nothing else in clusters/ currently uses won’t be seen at all. If clusters/ is completely empty, there’s nothing to derive a scan scope from and the scan finds nothing.

Caution

Not every module implements the scan — it depends on the driver’s own list.yaml operation (optional; a module without one simply doesn’t participate, contributing nothing to the scan rather than erroring). Check your specific driver module’s documentation.

To actually remove a cluster, either delete it explicitly (hyve cluster delete <name>, or spec.delete: true) or use scheduled expiry — see Deleting Clusters with onDelete Workflows below for why removing a cluster’s YAML file directly, without marking it deleted first, isn’t the recommended path either way.

Recommended Repository Layout

hyve-state/                    # Your Hyve state repository
├── hyve.yaml                  # Reconcile mode: cicd
├── hyve.lock                  # Locked module versions (commit this)
├── clusters/                  # Cluster definitions
│   ├── production.yaml
│   ├── staging.yaml
│   └── development.yaml
├── templates/                 # Reusable cluster templates
│   └── prod-template.yaml
└── workflows/                 # Lifecycle workflow definitions
    └── setup-monitoring.yaml

Example Cluster Definitions

# clusters/production.yaml — long-lived GKE cluster
apiVersion: v1
kind: Cluster
metadata:
  name: production
  region: us-central1
spec:
  driver:
    source: github.com/hyve-modules/gke
    version: v1.2.0
  params:
    project_id: my-gcp-project
    machine_type: e2-standard-4
    node_count: "3"

# clusters/dev-sprint-47.yaml — ephemeral cluster with automatic expiry
apiVersion: v1
kind: Cluster
metadata:
  name: dev-sprint-47
  region: PHX1
spec:
  driver:
    source: github.com/hyve-modules/civo
    version: v1.0.0
  params:
    node_size: g4s.kube.small
    node_count: "1"
  expiresAt: "2026-09-01T00:00:00Z"   # auto-deleted after this time
  workflows:
    onDelete: [notify-team]

# clusters/staging.yaml — temporarily paused during migration
apiVersion: v1
kind: Cluster
metadata:
  name: staging
  region: us-east-1
spec:
  driver:
    source: github.com/hyve-modules/eks
    version: v2.1.3
  params:
    vpc_id: vpc-0abc123456789
    eks_role_arn: arn:aws:iam::123456789012:role/eks-role
    node_role_arn: arn:aws:iam::123456789012:role/node-role
    instance_type: t3.medium
    node_count: "3"
  pause: true     # reconciler skips this cluster until removed

GitHub Actions Workflows

Separate State Repository

Use this pattern when your Hyve state is kept in a dedicated repository, separate from your application repositories.

hyve-state/                    # Dedicated state repository
├── .github/
│   └── workflows/
│       └── reconcile.yml
├── hyve.yaml                  # reconcile.mode: cicd
├── hyve.lock                  # Locked module versions
├── clusters/
│   ├── production.yaml
│   └── staging.yaml
├── templates/
│   └── prod-template.yaml
└── workflows/
    └── setup-monitoring.yaml

.github/workflows/reconcile.yml:

name: Hyve Reconcile

on:
  push:
    branches: [main]
    paths:
      - 'clusters/**'
      - 'templates/**'
      - 'workflows/**'
      - 'hyve.yaml'
      - 'hyve.lock'

  schedule:
    - cron: '0 20 * * *'   # Every day at 8:00 PM UTC

  workflow_dispatch:

jobs:
  reconcile:
    runs-on: ubuntu-latest
    permissions:
      contents: write   # Required to push reconciliation results back

    steps:
      - name: Checkout state repository
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITHUB_TOKEN }}

      - name: Configure git identity
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"

      - name: Install Hyve
        run: go install github.com/cbridges1/hyve@latest

      - name: Install modules
        run: hyve module install

      - name: Reconcile clusters
        env:
          # Credentials are read by modules from standard env vars.
          # Set whichever variables your modules expect.

          # Civo
          CIVO_TOKEN: ${{ secrets.CIVO_TOKEN }}

          # AWS
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_REGION: ${{ secrets.AWS_REGION }}

          # GCP
          GOOGLE_APPLICATION_CREDENTIALS: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}

          # Azure
          AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}
          AZURE_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }}
          AZURE_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }}
        run: hyve reconcile --path .

Credentials are passed as standard environment variables. Modules read them directly — the same env vars used by the cloud CLIs. Add whichever secrets your modules require as GitHub Actions secrets.

Scheduled Reconcile

Add a schedule trigger directly to your existing reconcile workflow. This is the backbone for all time-based cluster automation — expiry-based deletion (expiresAt), normal drift correction, and paused-cluster awareness all happen through the same reconcile path.

on:
  push:
    branches: [main]
    paths:
      - 'clusters/**'
      - 'hyve.yaml'
      - 'hyve.lock'

  schedule:
    - cron: '0 */1 * * *'  # Every hour — tightest window for expiresAt cluster cleanup

  workflow_dispatch:

Adjust the cron expression to match your preferred cadence. Common schedules:

Cron	Description
0 20 * * *	Daily at 8:00 PM UTC
0 * * * *	Hourly — tightest expiry window
0 0 * * *	Daily at midnight UTC
0 8,20 * * *	Twice daily at 8 AM and 8 PM UTC

Note

The git identity and contents: write permission are only exercised when a cluster is deleted (via spec.delete: true or an expired spec.expiresAt). They have no effect on create, update, or paused-cluster runs.

Full Teardown

To destroy every cluster in the state repository on a schedule (e.g. overnight cost savings), mark each one delete: true before reconciling — the same explicit, per-cluster mechanism Deleting Clusters with onDelete Workflows below covers, just applied in bulk. This is not what strictDelete is for: it only warns about clusters with no YAML at all, so removing every clusters/*.yaml file and relying on it to clean up the cloud side would do nothing but print warnings.

Caution

This workflow permanently deletes all clusters managed by the state repository.

.github/workflows/nightly-teardown.yml

name: Nightly Teardown

on:
  schedule:
    - cron: '0 0 * * *'   # Midnight UTC
  workflow_dispatch:

jobs:
  teardown:
    runs-on: ubuntu-latest
    permissions:
      contents: write

    steps:
      - uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITHUB_TOKEN }}

      - name: Configure git identity
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"

      - name: Install Hyve
        run: |
          curl -sSL https://github.com/cbridges1/hyve/releases/latest/download/hyve-linux-amd64 \
            -o /usr/local/bin/hyve
          chmod +x /usr/local/bin/hyve

      - name: Mark every cluster for deletion
        # spec.delete: true, not a deleted file — the reconciler needs the
        # YAML to still be there to know which onDelete workflows to run and
        # against which driver/params, exactly as in the mark-for-deletion
        # pattern below. It self-removes the file once teardown completes.
        run: |
          for f in clusters/*.yaml clusters/*.yml; do
            [ -f "$f" ] || continue
            yq -i '.spec.delete = true' "$f"
          done
          git add clusters/
          git commit -m "Nightly teardown: mark all clusters for deletion" || echo "Nothing to commit"
          git push

      - name: Reconcile (deletes every cluster marked above)
        env:
          PROD_US_AWS_ACCESS_KEY_ID: ${{ secrets.PROD_US_AWS_ACCESS_KEY_ID }}
          PROD_US_AWS_SECRET_ACCESS_KEY: ${{ secrets.PROD_US_AWS_SECRET_ACCESS_KEY }}
          MY_PROJECT_GCP_CREDENTIALS_JSON: ${{ secrets.MY_PROJECT_GCP_CREDENTIALS_JSON }}
          MY_ORG_CIVO_TOKEN: ${{ secrets.MY_ORG_CIVO_TOKEN }}
        run: hyve reconcile --path .

Deleting Clusters with onDelete Workflows

Deleting the YAML file for a cluster directly means the reconciler never sees the cluster definition — and therefore cannot run any onDelete workflows before the cluster is removed. To safely delete a cluster and have its lifecycle workflows execute, use the mark-for-deletion pattern instead.

How It Works

Set spec.delete: true in the cluster YAML without removing the file:

clusters/staging.yaml

apiVersion: v1
kind: Cluster
metadata:
  name: staging
spec:
  delete: true          # <-- marks this cluster for deletion
  provider: aws
  awsAccount: prod-us
  region: us-east-1
  clusterType: eks
  workflows:
    onDelete:
      - cleanup-namespaces
      - drain-nodes

When the reconciler processes this file it will:

Detect the deletion marker

DetermineAction returns ActionDelete as soon as it sees spec.delete: true — no cloud API call is needed to make this decision.

Run onDelete workflows

All workflows listed under spec.workflows.onDelete are executed against the cluster before any cloud resources are touched.

Delete the cluster

The cluster is removed from the cloud provider.

Delete the cluster

The cluster is removed from the cloud provider.

Remove the YAML file and push

The reconciler removes the cluster definition file from the repository, commits the change, and pushes it back to the remote. The deletion marker is never left behind.

Caution

Do not delete the YAML file directly if you need onDelete workflows to run. The reconciler must be able to read the file — including the workflow list — before it can execute lifecycle hooks.

Required GitHub Actions Configuration

Because the reconciler removes the cluster file and pushes back to the repository after deletion, the GitHub Actions workflow needs write access to the repository and a valid git committer identity.

1. Contents write permission

jobs:
  reconcile:
    runs-on: ubuntu-latest
    permissions:
      contents: write

2. Git committer identity

GitHub Actions runners do not configure a git identity by default. Add this step before running Hyve:

- name: Configure git identity
  run: |
    git config user.name "github-actions[bot]"
    git config user.email "github-actions[bot]@users.noreply.github.com"

3. Branch protection rules

If your default branch requires pull request reviews or passing checks before merging, the reconciler’s direct push will be rejected.

Choose one of:

• Exempt the Actions bot — add github-actions[bot] to the bypass list in your branch protection settings.

• Use a Personal Access Token (PAT) — pass it to the checkout step so the push is made as a user with bypass permissions.

  - uses: actions/checkout@v4
    with:
      token: ${{ secrets.HYVE_PAT }}

Complete example

name: Hyve Reconcile

on:
  push:
    branches: [main]
    paths:
      - 'clusters/**'
      - 'provider-configs/**'
      - 'hyve.yaml'

  schedule:
    - cron: '0 20 * * *'   # Every day at 8:00 PM UTC

  workflow_dispatch:

jobs:
  reconcile:
    runs-on: ubuntu-latest
    permissions:
      contents: write

    steps:
      - name: Checkout state repository
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITHUB_TOKEN }}

      - name: Configure git identity
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"

      - name: Install Hyve
        run: |
          curl -sSL https://github.com/cbridges1/hyve/releases/latest/download/hyve-linux-amd64 \
            -o /usr/local/bin/hyve
          chmod +x /usr/local/bin/hyve

      - name: Reconcile clusters
        env:
          PROD_US_AWS_ACCESS_KEY_ID: ${{ secrets.PROD_US_AWS_ACCESS_KEY_ID }}
          PROD_US_AWS_SECRET_ACCESS_KEY: ${{ secrets.PROD_US_AWS_SECRET_ACCESS_KEY }}
        run: hyve reconcile --path .

Note

The Configure git identity step and contents: write permission are only exercised when a cluster marked spec.delete: true is processed. They have no effect on normal create or update reconciliations.

Pausing Reconciliation

Set spec.pause: true to keep a cluster’s definition in the repository while telling the reconciler to skip it entirely. The cluster continues running in the cloud — Hyve simply does not compare desired state or make any changes until pause is removed.

clusters/staging.yaml

apiVersion: v1
kind: Cluster
metadata:
  name: staging
spec:
  pause: true       # reconciler skips this cluster on every run
  provider: civo
  region: NYC1
  nodes: [g4s.kube.medium]

Common use cases:

• Maintenance windows — prevent Hyve from touching a cluster while you perform manual operations
• Cost investigation — keep the definition versioned while you decide whether to delete or resize
• Temporary divergence — allow a cluster to drift from its YAML definition without triggering an update

To resume normal reconciliation, remove the pause: true line (or set it to false) and commit.

Caution

A paused cluster’s YAML is still present in clusters/, so the strictDelete provider scan never flags it as untracked. Pause does not affect cloud resources — it only affects Hyve’s behavior.

---

Scheduled Cluster Expiry

Set spec.expiresAt to an RFC 3339 timestamp to have the reconciler automatically delete a cluster when that time passes. When the scheduled reconcile runs after the expiry time, the reconciler treats the cluster exactly as if spec.delete: true had been set — running onDelete workflows, deleting from the cloud provider, and removing the YAML file.

clusters/dev-sprint-47.yaml

apiVersion: v1
kind: Cluster
metadata:
  name: dev-sprint-47
spec:
  provider: civo
  region: PHX1
  nodes: [g4s.kube.small]
  expiresAt: "2026-05-01T00:00:00Z"   # deleted automatically after this time
  workflows:
    onDelete:
      - notify-team
      - cleanup-namespaces

The expiry is checked on every reconcile run. With an hourly scheduled reconcile, clusters are cleaned up within ~1 hour of their deadline.

Combining pause and expiresAt:

A paused cluster is always skipped — the expiry check is never reached. If you pause a cluster that has an expiresAt set, it will not be deleted until pause is removed.

Scheduled Reconcile Workflow

No special pipeline logic is needed. The schedule trigger on your existing reconcile workflow handles drift correction, expirations, and deletions in a single run — no separate workflow file required.

The expiry check runs on every reconcile. With a daily schedule at 8 PM, expired clusters are cleaned up within 24 hours of their deadline. Use an hourly schedule (0 * * * *) if you need tighter expiry windows.

Ephemeral Environment Pattern

A common pattern for feature-branch or sprint clusters:

clusters/feature-login-redesign.yaml

apiVersion: v1
kind: Cluster
metadata:
  name: feature-login-redesign
spec:
  provider: aws
  awsAccount: dev
  region: us-east-1
  clusterType: eks
  expiresAt: "2026-04-25T18:00:00Z"   # end of sprint
  workflows:
    onCreate:
      - deploy-feature-branch
    onDelete:
      - export-test-results
      - notify-slack

When the sprint ends, the cluster tears itself down automatically — no manual cleanup required.

---

End-to-End Workflow

Here is the complete flow from developer change to provisioned cluster:

# 1. Developer: add a new cluster definition locally
hyve cluster add staging \
  --provider gcp \
  --project-name my-project \
  --region us-central1 \
  --nodes e2-standard-4,e2-standard-4

# 2. Developer: run reconcile (in cicd mode — pushes state, does NOT provision)
hyve reconcile
# Output:
# Reconcile mode: cicd
# Skipping local reconciliation — cluster provisioning will be handled by the CI/CD pipeline.
# Pushing desired state to repository...
# ✅ Desired state pushed to repository. The CI/CD pipeline will reconcile.

# 3. GitHub Actions triggers on push to main
# 4. Pipeline runs: hyve reconcile --path .
# Output:
# Using local repository path: /home/runner/work/hyve-state/hyve-state
# Cluster 'staging' created in GCP (us-central1)
# ✅ Changes committed and pushed to remote repository successfully
# Cluster reconciliation completed

Credential Configuration

Modules read credentials from environment variables — the same env vars used by their underlying cloud CLIs. Set them as GitHub Actions secrets.

Standard credential environment variables

Provider	Environment variables
Civo	CIVO_TOKEN
AWS	AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION
GCP	GOOGLE_APPLICATION_CREDENTIALS (path to service account JSON), or use OIDC
Azure	AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET

You can also pass credentials as module params via cluster.spec.params, but environment variables are preferred for CI/CD since they are never written to Git.

Required GitHub Secrets

Set these in your repository’s Settings → Secrets and variables → Actions.

Secret	Description
GITHUB_TOKEN	Automatically provided by GitHub Actions; needs contents: write permission
Provider credentials	Add whichever credential env vars your modules expect (see table above)

Caution

Never commit literal API tokens or cloud credentials to your repository. Store them as GitHub Actions secrets and inject them as environment variables in your reconcile workflow.

Security Considerations

Use OIDC for AWS and GCP

Instead of storing long-lived AWS/GCP credentials as secrets, use GitHub Actions’ OIDC provider with AWS IAM roles or GCP Workload Identity Federation. This eliminates the need to rotate secrets.

permissions:
id-token: write
contents: read

Limit repository permissions

Grant the contents: write permission only to the reconcile workflow, not globally. Use branch protection rules to require PR reviews before merging cluster changes to main.

Audit trail through Git history

Because all state changes flow through Git, every cluster change has an associated commit, author, and timestamp. Use git log clusters/ to see the full history.

Review changes before they apply

Use pull requests for cluster changes. Engineers open a PR modifying clusters/*.yaml; reviewers approve; the merge to main triggers the pipeline. This gives you a review gate before anything is provisioned.

Troubleshooting

Pipeline fails: Civo API token not found

Check that provider-configs/civo.yaml has a token field pointing to an env var reference, and that the corresponding secret is available in the workflow:

provider-configs/civo.yaml

organizations:
  - name: my-org
    org_id: cbridges1-id
    token: ${MY_ORG_CIVO_TOKEN}

# workflow env:
MY_ORG_CIVO_TOKEN: ${{ secrets.MY_ORG_CIVO_TOKEN }}

Pipeline fails: permission denied pushing to repository

Ensure the workflow has contents: write permission:

permissions:
contents: write

Or use a token with write access in the checkout step.

Local hyve reconcile provisions instead of pushing

Check that hyve.yaml is committed to the root of your state repository with reconcile.mode: cicd. If the file is absent, Hyve defaults to local mode.

onDelete workflows did not run before cluster was deleted

This happens when the cluster YAML file was deleted directly from the repository instead of using the spec.delete: true marker. The reconciler never saw the file, so it had no workflow list to execute — the cluster appeared as an unmanaged orphan and was removed without running lifecycle hooks.

Prevention: Always set spec.delete: true in the YAML file and let the reconciler handle the deletion. Do not git rm or manually delete cluster files when onDelete workflows need to run.

Pipeline fails: permission denied pushing cluster file removal

The reconciler needs write access to commit the cluster file deletion after spec.delete: true is processed.

Ensure the workflow has contents: write permission and a git identity is configured:

permissions:
  contents: write

steps:
  - name: Configure git identity
    run: |
      git config user.name "github-actions[bot]"
      git config user.email "github-actions[bot]@users.noreply.github.com"

If branch protection rules block direct pushes, see Branch protection rules.

hyve reconcile --path . fails: no clusters found

Verify that cluster YAML files exist under a clusters/ directory at the path you provided. The --path flag should point to the root of the repository, not the clusters/ subdirectory.

Cluster with expiresAt was not deleted on schedule

The expiry check runs on every reconcile. Verify:

1. The scheduled reconcile workflow is running — check the Actions tab in GitHub.
2. The expiresAt value is a valid RFC 3339 timestamp: "2026-05-01T00:00:00Z". An invalid format logs a warning and skips the expiry check.
3. The cluster does not also have pause: true set — a paused cluster is skipped entirely before the expiry check is reached.
4. The workflow has contents: write permission and a git identity configured, since the reconciler needs to commit the YAML file removal after deletion.

Paused cluster is being modified by the reconciler

If a cluster with pause: true is still being reconciled, verify that the change has been committed and pushed to the branch the pipeline checks out. The reconciler reads the YAML files from the checked-out repository — a local change that has not been pushed will not take effect in the pipeline.

Related

Configuration

Authentication and provider setup

Git Management

Configure Git repositories

Cluster CLI

Full cluster command reference

Quick Start

Get up and running quickly