CI/CD Workflow
Overview
Hyve’s CI/CD mode separates the desired state (what you want) from the actual provisioning (making it happen). Engineers declare infrastructure changes locally; a pipeline running in your CI/CD system performs the cloud operations. This keeps credentials off developer machines and provides a full audit trail through Git history.
Local Mode
Default. hyve reconcile provisions clusters directly from the machine that runs the command. Good for local development and simple setups.
CI/CD Mode
hyve reconcile validates desired state and exits without provisioning — you push it, and a pipeline calls hyve reconcile --path . to do the actual provisioning. Credentials never leave the pipeline.
How It Works
Engineer declares a change
A team member adds or modifies a cluster definition and runs hyve reconcile locally.
Hyve validates and exits — you commit and push
In cicd mode, Hyve validates the YAML and exits without touching any cloud APIs. Hyve never commits or pushes on its own — see Version Control — so committing and pushing the desired state is your own explicit step: git commit -am "..." && git push.
Pipeline triggers
The push triggers a CI/CD workflow (e.g. GitHub Actions).
Pipeline reconciles
The pipeline checks out the repository and runs hyve reconcile --path .. Because --path is provided, Hyve always runs full local reconciliation regardless of the hyve.yaml mode setting.
Commit results back — your own step
Hyve writes reconciliation results (cluster status, driver outputs) back to local files. Committing and pushing them is a pipeline step you add explicitly (git add -A clusters cluster-state && git commit -m "hyve: reconcile" && git push), not something hyve does for you.
Repository Setup
Enable CI/CD Mode
Add a hyve.yaml file to the root of your Hyve state repository:
reconcile: mode: cicdThis single file tells Hyve to skip local provisioning when hyve reconcile is called without --path.
Provider Scan (strictDelete)
By default, Hyve only ever acts on clusters that have a YAML definition in clusters/ — a cloud cluster it doesn’t know about is invisible to it, full stop.
Enable strictDelete to have Hyve additionally scan the cloud provider after each reconcile and warn about any cluster it finds that has no matching clusters/*.yaml:
reconcile: mode: cicd strictDelete: true[strictDelete] Scanning 1 driver/region pair(s) for untracked clusters...[strictDelete] Warning: cluster "leftover-demo" exists in ./custom-modules/civo@latest region=NYC1 but is not tracked in clusters/ — hyve will not touch it automatically; delete it manually if it shouldn't exist, or add a clusters/leftover-demo.yaml if it should be tracked[strictDelete] Scan complete — 1 untracked cluster(s) foundThis is deliberately warn-only — it never deletes anything. A cluster is too large a blast radius to remove automatically off a provider scan, especially since the scan itself can have blind spots: it only queries the driver module + region combinations already represented by some tracked cluster (there’s no standalone “every region this account has ever used” concept), so a cluster using a driver/region combination nothing else in clusters/ currently uses won’t be seen at all. If clusters/ is completely empty, there’s nothing to derive a scan scope from and the scan finds nothing.
To actually remove a cluster, either delete it explicitly (hyve cluster delete <name>, or spec.delete: true) or use scheduled expiry — see Deleting Clusters with onDelete Workflows below for why removing a cluster’s YAML file directly, without marking it deleted first, isn’t the recommended path either way.
Recommended Repository Layout
hyve-state/ # Your Hyve state repository├── hyve.yaml # Reconcile mode: cicd├── hyve.lock # Locked module versions (commit this)├── clusters/ # Cluster definitions│ ├── production.yaml│ ├── staging.yaml│ └── development.yaml├── templates/ # Reusable cluster templates│ └── prod-template.yaml└── workflows/ # Lifecycle workflow definitions └── setup-monitoring.yamlExample Cluster Definitions
# clusters/production.yaml — long-lived GKE clusterapiVersion: hyve.io/v1alpha1kind: ClusterDefinitionmetadata: name: productionspec: region: us-central1 driver: source: github.com/your-org/hyve-gke-module version: v1.2.0 params: project_id: my-gcp-project machine_type: e2-standard-4 node_count: "3"# clusters/dev-sprint-47.yaml — ephemeral cluster with automatic expiryapiVersion: hyve.io/v1alpha1kind: ClusterDefinitionmetadata: name: dev-sprint-47spec: region: PHX1 driver: source: github.com/your-org/hyve-civo-module version: v1.0.0 params: node_size: g4s.kube.small node_count: "1" expiresAt: "2026-09-01T00:00:00Z" # auto-deleted after this time workflows: onDelete: [notify-team]# clusters/staging.yaml — temporarily paused during migrationapiVersion: hyve.io/v1alpha1kind: ClusterDefinitionmetadata: name: stagingspec: region: us-east-1 driver: source: github.com/your-org/hyve-eks-module version: v2.1.3 params: vpc_id: vpc-0abc123456789 eks_role_arn: arn:aws:iam::123456789012:role/eks-role node_role_arn: arn:aws:iam::123456789012:role/node-role instance_type: t3.medium node_count: "3" pause: true # reconciler skips this cluster until removedGitHub Actions Workflows
Separate State Repository
Use this pattern when your Hyve state is kept in a dedicated repository, separate from your application repositories.
hyve-state/ # Dedicated state repository├── .github/│ └── workflows/│ └── reconcile.yml├── hyve.yaml # reconcile.mode: cicd├── hyve.lock # Locked module versions├── clusters/│ ├── production.yaml│ └── staging.yaml├── templates/│ └── prod-template.yaml└── workflows/ └── setup-monitoring.yaml.github/workflows/reconcile.yml:
name: Hyve Reconcile
on: push: branches: [main] paths: - 'clusters/**' - 'templates/**' - 'workflows/**' - 'hyve.yaml' - 'hyve.lock'
schedule: - cron: '0 20 * * *' # Every day at 8:00 PM UTC
workflow_dispatch:
jobs: reconcile: runs-on: ubuntu-latest permissions: contents: write # Required to push reconciliation results back
steps: - name: Checkout state repository uses: actions/checkout@v4 with: token: ${{ secrets.GITHUB_TOKEN }}
- name: Configure git identity run: | git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com"
- uses: actions/setup-go@v5 with: go-version: "1.26"
- name: Install Hyve run: | go install github.com/cbridges1/hyve@latest echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH"
- name: Install modules run: hyve module install
- name: Reconcile clusters env: # Credentials are read by modules from standard env vars. # Set whichever variables your modules expect.
# Civo CIVO_TOKEN: ${{ secrets.CIVO_TOKEN }}
# AWS AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} AWS_REGION: ${{ secrets.AWS_REGION }}
# GCP GOOGLE_APPLICATION_CREDENTIALS: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}
# Azure AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }} AZURE_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }} AZURE_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }} run: hyve reconcile --path .
- name: Commit and push state # hyve never commits or pushes — this records whatever reconcile # changed: cluster-state/ sidecars, and the files of clusters it # just deleted. if: always() keeps partial progress. if: always() run: | git add -A clusters cluster-state git diff --cached --quiet || { git commit -m "hyve: reconcile" && git push; }Credentials are passed as standard environment variables. Modules read them directly — the same env vars used by the cloud CLIs. Add whichever secrets your modules require as GitHub Actions secrets.
Scheduled Reconcile
Add a schedule trigger directly to your existing reconcile workflow. This is the backbone for all time-based cluster automation — expiry-based deletion (expiresAt), normal drift correction, and paused-cluster awareness all happen through the same reconcile path.
on: push: branches: [main] paths: - 'clusters/**' - 'hyve.yaml' - 'hyve.lock'
schedule: - cron: '0 */1 * * *' # Every hour — tightest window for expiresAt cluster cleanup
workflow_dispatch:Adjust the cron expression to match your preferred cadence. Common schedules:
| Cron | Description |
|---|---|
0 20 * * * | Daily at 8:00 PM UTC |
0 * * * * | Hourly — tightest expiry window |
0 0 * * * | Daily at midnight UTC |
0 8,20 * * * | Twice daily at 8 AM and 8 PM UTC |
Full Teardown
To destroy every cluster in the state repository on a schedule (e.g. overnight cost savings), mark each one delete: true before reconciling — the same explicit, per-cluster mechanism Deleting Clusters with onDelete Workflows below covers, just applied in bulk. This is not what strictDelete is for: it only warns about clusters with no YAML at all, so removing every clusters/*.yaml file and relying on it to clean up the cloud side would do nothing but print warnings.
name: Nightly Teardown
on: schedule: - cron: '0 0 * * *' # Midnight UTC workflow_dispatch:
jobs: teardown: runs-on: ubuntu-latest permissions: contents: write
steps: - uses: actions/checkout@v4 with: token: ${{ secrets.GITHUB_TOKEN }}
- name: Configure git identity run: | git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com"
- uses: actions/setup-go@v5 with: go-version: "1.26"
- name: Install Hyve run: | go install github.com/cbridges1/hyve@latest echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH"
- name: Mark every cluster for deletion # spec.delete: true, not a deleted file — the reconciler needs the # YAML to still be there to know which onDelete workflows to run and # against which driver/params, exactly as in the mark-for-deletion # pattern below. It removes the file locally once teardown # completes; the commit step at the end pushes that removal. run: | for f in clusters/*.yaml clusters/*.yml; do [ -f "$f" ] || continue yq -i '.spec.delete = true' "$f" done git add clusters/ git commit -m "Nightly teardown: mark all clusters for deletion" || echo "Nothing to commit" git push
- name: Reconcile (deletes every cluster marked above) env: PROD_US_AWS_ACCESS_KEY_ID: ${{ secrets.PROD_US_AWS_ACCESS_KEY_ID }} PROD_US_AWS_SECRET_ACCESS_KEY: ${{ secrets.PROD_US_AWS_SECRET_ACCESS_KEY }} MY_PROJECT_GCP_CREDENTIALS_JSON: ${{ secrets.MY_PROJECT_GCP_CREDENTIALS_JSON }} MY_ORG_CIVO_TOKEN: ${{ secrets.MY_ORG_CIVO_TOKEN }} run: hyve reconcile --path .
- name: Commit and push state # hyve never commits or pushes — this records whatever reconcile # changed: cluster-state/ sidecars, and the files of clusters it # just deleted. if: always() keeps partial progress. if: always() run: | git add -A clusters cluster-state git diff --cached --quiet || { git commit -m "hyve: reconcile" && git push; }Deleting Clusters with onDelete Workflows
Deleting the YAML file for a cluster directly means the reconciler never sees the cluster definition — and therefore cannot run any onDelete workflows before the cluster is removed. To safely delete a cluster and have its lifecycle workflows execute, use the mark-for-deletion pattern instead.
How It Works
Set spec.delete: true in the cluster YAML without removing the file:
apiVersion: hyve.io/v1alpha1kind: ClusterDefinitionmetadata: name: stagingspec: delete: true # <-- marks this cluster for deletion driver: source: github.com/your-org/hyve-eks-module version: v2.1.3 region: us-east-1 params: instance_type: t3.medium node_count: "3" workflows: onDelete: - cleanup-namespaces - drain-nodesWhen the reconciler processes this file it will:
Detect the deletion marker
DetermineAction returns ActionDelete as soon as it sees spec.delete: true — no cloud API call is needed to make this decision.
Run onDelete workflows
All workflows listed under spec.workflows.onDelete are executed against the cluster before any cloud resources are touched.
Delete the cluster
The cluster is removed from the cloud provider.
Delete the cluster
The cluster is removed from the cloud provider.
Remove the YAML file and push
The reconciler removes the cluster definition file from the repository, commits the change, and pushes it back to the remote. The deletion marker is never left behind.
Required GitHub Actions Configuration
The reconciler removes a deleted cluster’s file (and its cluster-state/ sidecar) locally; your pipeline’s commit step pushes that back. So the workflow needs write access to the repository and a git committer identity.
1. Contents write permission
jobs: reconcile: runs-on: ubuntu-latest permissions: contents: write2. Git committer identity
GitHub Actions runners do not configure a git identity by default. Add this step before the commit step:
- name: Configure git identity run: | git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com"3. Branch protection rules
If your default branch requires pull request reviews or passing checks before merging, the pipeline’s direct push will be rejected.
Choose one of:
-
Exempt the Actions bot — add
github-actions[bot]to the bypass list in your branch protection settings. -
Use a Personal Access Token (PAT) — pass it to the checkout step so the push is made as a user with bypass permissions.
- uses: actions/checkout@v4with:token: ${{ secrets.HYVE_PAT }}
Complete example
name: Hyve Reconcile
on: push: branches: [main] paths: - 'clusters/**' - 'templates/**' - 'workflows/**'
schedule: - cron: '0 20 * * *' # Every day at 8:00 PM UTC
workflow_dispatch:
jobs: reconcile: runs-on: ubuntu-latest permissions: contents: write
steps: - name: Checkout state repository uses: actions/checkout@v4 with: token: ${{ secrets.GITHUB_TOKEN }}
- name: Configure git identity run: | git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com"
- uses: actions/setup-go@v5 with: go-version: "1.26"
- name: Install Hyve run: | go install github.com/cbridges1/hyve@latest echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH"
- name: Reconcile clusters env: PROD_US_AWS_ACCESS_KEY_ID: ${{ secrets.PROD_US_AWS_ACCESS_KEY_ID }} PROD_US_AWS_SECRET_ACCESS_KEY: ${{ secrets.PROD_US_AWS_SECRET_ACCESS_KEY }} run: hyve reconcile --path .
- name: Commit and push state # hyve never commits or pushes — this records whatever reconcile # changed: cluster-state/ sidecars, and the files of clusters it # just deleted. if: always() keeps partial progress. if: always() run: | git add -A clusters cluster-state git diff --cached --quiet || { git commit -m "hyve: reconcile" && git push; }Pausing Reconciliation
Set spec.pause: true to keep a cluster’s definition in the repository while telling the reconciler to skip it entirely. The cluster continues running in the cloud — Hyve simply does not compare desired state or make any changes until pause is removed.
apiVersion: hyve.io/v1alpha1kind: ClusterDefinitionmetadata: name: stagingspec: pause: true # reconciler skips this cluster on every run driver: source: github.com/your-org/hyve-civo-module version: v1.0.0 region: NYC1 params: node_size: g4s.kube.mediumCommon use cases:
- Maintenance windows — prevent Hyve from touching a cluster while you perform manual operations
- Cost investigation — keep the definition versioned while you decide whether to delete or resize
- Temporary divergence — allow a cluster to drift from its YAML definition without triggering an update
To resume normal reconciliation, remove the pause: true line (or set it to false) and commit.
Scheduled Cluster Expiry
Set spec.expiresAt to an RFC 3339 timestamp to have the reconciler automatically delete a cluster when that time passes. When the scheduled reconcile runs after the expiry time, the reconciler treats the cluster exactly as if spec.delete: true had been set — running onDelete workflows, deleting from the cloud provider, and removing the YAML file.
apiVersion: hyve.io/v1alpha1kind: ClusterDefinitionmetadata: name: dev-sprint-47spec: driver: source: github.com/your-org/hyve-civo-module version: v1.0.0 region: PHX1 params: node_size: g4s.kube.small expiresAt: "2026-05-01T00:00:00Z" # deleted automatically after this time workflows: onDelete: - notify-team - cleanup-namespacesThe expiry is checked on every reconcile run. With an hourly scheduled reconcile, clusters are cleaned up within ~1 hour of their deadline.
Combining pause and expiresAt:
A paused cluster is always skipped — the expiry check is never reached. If you pause a cluster that has an expiresAt set, it will not be deleted until pause is removed.
Scheduled Reconcile Workflow
No special pipeline logic is needed. The schedule trigger on your existing reconcile workflow handles drift correction, expirations, and deletions in a single run — no separate workflow file required.
The expiry check runs on every reconcile. With a daily schedule at 8 PM, expired clusters are cleaned up within 24 hours of their deadline. Use an hourly schedule (0 * * * *) if you need tighter expiry windows.
Ephemeral Environment Pattern
A common pattern for feature-branch or sprint clusters:
apiVersion: hyve.io/v1alpha1kind: ClusterDefinitionmetadata: name: feature-login-redesignspec: driver: source: github.com/your-org/hyve-eks-module version: v2.1.3 region: us-east-1 params: instance_type: t3.medium node_count: "2" expiresAt: "2026-04-25T18:00:00Z" # end of sprint workflows: onCreate: - deploy-feature-branch onDelete: - export-test-results - notify-slackWhen the sprint ends, the cluster tears itself down automatically — no manual cleanup required.
End-to-End Workflow
Here is the complete flow from developer change to provisioned cluster:
# 1. Developer: create a new cluster definition locally, from a templatehyve cluster create staging --template gcp-standard \ --region us-central1 --set node_count=2
# 2. Developer: run reconcile (in cicd mode — validates, does NOT provision)hyve reconcile# Output:# Reconcile mode: cicd# Skipping local reconciliation — cluster provisioning will be handled by the CI/CD pipeline.# 💡 Push the desired state yourself so the CI/CD pipeline picks it up: git add -A && git commit -m ... && git push
# 3. Developer: push itgit add clusters/staging.yaml && git commit -m "Add staging" && git push
# 4. GitHub Actions triggers on push to main and runs: hyve reconcile --path .# Cluster 'staging' is created in GCP (us-central1)# 5. The pipeline's commit step pushes cluster-state/staging.state.yaml backCredential Configuration
Modules read credentials from environment variables — the same env vars used by their underlying cloud CLIs. Set them as GitHub Actions secrets.
Standard credential environment variables
| Provider | Environment variables |
|---|---|
| Civo | CIVO_TOKEN |
| AWS | AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION |
| GCP | GOOGLE_APPLICATION_CREDENTIALS (path to service account JSON), or use OIDC |
| Azure | AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET |
You can also pass credentials as module params via cluster.spec.params, but environment variables are preferred for CI/CD since they are never written to Git.
Required GitHub Secrets
Set these in your repository’s Settings → Secrets and variables → Actions.
| Secret | Description |
|---|---|
GITHUB_TOKEN | Automatically provided by GitHub Actions; needs contents: write permission |
| Provider credentials | Add whichever credential env vars your modules expect (see table above) |
Security Considerations
Use OIDC for AWS and GCP
Instead of storing long-lived AWS/GCP credentials as secrets, use GitHub Actions’ OIDC provider with AWS IAM roles or GCP Workload Identity Federation. This eliminates the need to rotate secrets.
permissions:id-token: writecontents: readLimit repository permissions
Grant the contents: write permission only to the reconcile workflow, not globally. Use branch protection rules to require PR reviews before merging cluster changes to main.
Audit trail through Git history
Because all state changes flow through Git, every cluster change has an associated commit, author, and timestamp. Use git log clusters/ to see the full history.
Review changes before they apply
Use pull requests for cluster changes. Engineers open a PR modifying clusters/*.yaml; reviewers approve; the merge to main triggers the pipeline. This gives you a review gate before anything is provisioned.
Troubleshooting
Pipeline fails: Civo API token not found
Hyve has no built-in provider-credentials mechanism — each module reads whatever
environment variables its own driver expects (for the Civo module, typically
CIVO_TOKEN). Make sure the CI job exports that variable from a secret before
running hyve:
# workflow env:env: CIVO_TOKEN: ${{ secrets.MY_ORG_CIVO_TOKEN }}Pipeline fails: permission denied pushing to repository
Ensure the workflow has contents: write permission:
permissions:contents: writeOr use a token with write access in the checkout step.
Local hyve reconcile provisions instead of pushing
Check that hyve.yaml is committed to the root of your state repository with reconcile.mode: cicd. If the file is absent, Hyve defaults to local mode.
onDelete workflows did not run before cluster was deleted
This happens when the cluster YAML file was deleted directly from the repository instead of using the spec.delete: true marker. The reconciler never saw the file, so it had no workflow list to execute — the cluster appeared as an unmanaged orphan and was removed without running lifecycle hooks.
Prevention: Always set spec.delete: true in the YAML file and let the reconciler handle the deletion. Do not git rm or manually delete cluster files when onDelete workflows need to run.
Pipeline fails: permission denied pushing cluster file removal
The pipeline’s commit step needs write access to push the cluster file removal after spec.delete: true is processed.
Ensure the workflow has contents: write permission and a git identity is configured:
permissions: contents: write
steps: - name: Configure git identity run: | git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com"If branch protection rules block direct pushes, see Branch protection rules.
hyve reconcile --path . fails: no clusters found
Verify that cluster YAML files exist under a clusters/ directory at the path you provided. The --path flag should point to the root of the repository, not the clusters/ subdirectory.
Cluster with expiresAt was not deleted on schedule
The expiry check runs on every reconcile. Verify:
- The scheduled reconcile workflow is running — check the Actions tab in GitHub.
- The
expiresAtvalue is a valid RFC 3339 timestamp:"2026-05-01T00:00:00Z". An invalid format logs a warning and skips the expiry check. - The cluster does not also have
pause: trueset — a paused cluster is skipped entirely before the expiry check is reached. - The workflow has
contents: write, a git identity, and a commit step, so the YAML file removal after deletion is pushed back.
Paused cluster is being modified by the reconciler
If a cluster with pause: true is still being reconciled, verify that the change has been committed and pushed to the branch the pipeline checks out. The reconciler reads the YAML files from the checked-out repository — a local change that has not been pushed will not take effect in the pipeline.