Published on

🚀 ArcanaAI CI/CD: A Hands-On GitOps Deployment Tutorial

Authors

🚀 ArcanaAI CI/CD: A Hands-On GitOps Deployment Tutorial

This is the reproducible, command-by-command account of ArcanaAI's production delivery work. It explains which machine runs each command, what the command proves, and where the safety boundary is.

The finished chain is:

ArcanaAI commit
  -> GitHub Actions tests and builds immutable SHA-tagged images
  -> arcana-deployment records those exact image tags
  -> Argo CD renders and compares desired state
  -> a reviewed sync applies it to K3s on the VPS
  -> Traefik and Cloudflare publish only intended endpoints

All values below are sanitized. Replace placeholders such as <OWNER>, <DOMAIN>, <VPS_ADDRESS>, <GIT_SHA>, and <SSH_USER>. Never commit or paste private keys, age keys, tunnel tokens, API keys, or passwords.

1. Repositories and execution contexts

~/Personal/arcana-ai          application source, Dockerfiles, CI workflow
~/Personal/arcana-deployment   Kubernetes manifests and encrypted configuration

Use a MacBook or company Ubuntu laptop for administration; the machine does not matter as long as it can SSH to the VPS. The VPS runs K3s. Cloudflare configuration is performed in the browser.

2. Create the private Kubernetes access path

The SSH config contains an alias similar to this:

Host vps
    HostName <VPS_ADDRESS>
    User <SSH_USER>
    IdentityFile ~/.ssh/<vps-key>

Keep this API tunnel open in one terminal:

ssh -N -L 16443:127.0.0.1:6443 vps

Create a dedicated kubeconfig and lock its permissions:

mkdir -p "$HOME/.kube"
chmod 700 "$HOME/.kube"
export KUBECONFIG="$HOME/.kube/arcana-k3s.yaml"
chmod 600 "$KUBECONFIG"

# macOS:
test "$(stat -f '%Lp' "$KUBECONFIG")" = 600

# Ubuntu/Linux:
test "$(stat -c '%a' "$KUBECONFIG")" = 600

The server in this file is 127.0.0.1:16443, reached through SSH. Verify:

export KUBECONFIG="$HOME/.kube/arcana-k3s.yaml"
kubectl cluster-info
kubectl get nodes -o wide
kubectl get pods -A

If kubectl tries localhost:8080, KUBECONFIG was not exported in that shell. Re-export it; do not troubleshoot the cluster until this is corrected.

3. Preflight K3s and the existing edge

Run on the workstation:

ssh vps 'sudo systemctl is-active k3s'
ssh vps 'sudo k3s --version'
ssh vps 'sudo k3s kubectl get nodes -o wide'
ssh vps 'sudo k3s kubectl get pods -A'
ssh vps 'sudo k3s kubectl get svc -A'
ssh vps 'sudo ss -lntup | grep -E ":(80|443|6443)\b"'
ssh vps 'docker ps --filter name=traefik --format "table {{.Names}}\t{{.Image}}\t{{.Ports}}"'
ssh vps 'sudo ufw status | grep 6443 || true'
ssh vps 'df -h /'
ssh vps 'sudo du -sh /var/lib/rancher'

Acceptance criteria: K3s is active, the node is Ready, system pods become Running, Traefik keeps 80/443, 6443 is reached through SSH, and disk usage is recorded. Let initial pods settle:

kubectl get pods -A -o wide
kubectl get daemonsets,deployments -A

4. Install Argo CD at a pinned version

ARGOCD_MANIFEST=/tmp/argocd-install-v3.4.5.yaml
if test -e "$ARGOCD_MANIFEST"; then
  echo "Refusing to overwrite $ARGOCD_MANIFEST" >&2
  exit 1
fi

curl --proto '=https' --tlsv1.2 -fsSL \
  https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.5/manifests/install.yaml \
  -o "$ARGOCD_MANIFEST"
test -s "$ARGOCD_MANIFEST"
sha256sum "$ARGOCD_MANIFEST"
rg -n '^kind: (CustomResourceDefinition|ClusterRole|ClusterRoleBinding|Deployment|StatefulSet|Service)$' \
  "$ARGOCD_MANIFEST"

export KUBECONFIG="$HOME/.kube/arcana-k3s.yaml"
kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts -f "$ARGOCD_MANIFEST"
kubectl wait -n argocd --for=condition=Available deployment --all --timeout=5m
kubectl rollout status -n argocd statefulset/argocd-application-controller --timeout=5m
kubectl get pods -n argocd -o wide
kubectl get svc -n argocd

Argo CD Redis is an internal Argo service. ArcanaAI Redis is a separate application dependency.

5. Register the Git repository

Generate a read-only deploy key:

ssh-keygen -t ed25519 \
  -f "$HOME/.ssh/arcana-deployment-argocd" \
  -C "argocd-readonly-arcana-deployment"
chmod 600 "$HOME/.ssh/arcana-deployment-argocd"
cat "$HOME/.ssh/arcana-deployment-argocd.pub"

Add the public key to GitHub with write access disabled, then test:

ssh-keygen -F github.com >/dev/null || {
  echo 'github.com is missing from known_hosts; verify its host key first.' >&2
  exit 1
}
ARGOCD_REPO_KEY="$HOME/.ssh/arcana-deployment-argocd"
GIT_SSH_COMMAND="ssh -i $ARGOCD_REPO_KEY -o IdentitiesOnly=yes" \
  git ls-remote git@github.com:<OWNER>/arcana-deployment.git HEAD

Create and label the Argo CD repository Secret without printing values:

if kubectl get secret -n argocd arcana-deployment-repo >/dev/null 2>&1; then
  echo 'Refusing to replace the existing repository Secret' >&2
  exit 1
fi
kubectl create secret generic arcana-deployment-repo \
  --namespace=argocd \
  --from-literal=type=git \
  --from-literal=url=git@github.com:<OWNER>/arcana-deployment.git \
  --from-literal=project=default \
  --from-file=sshPrivateKey="$ARGOCD_REPO_KEY"
kubectl label secret arcana-deployment-repo --namespace=argocd \
  argocd.argoproj.io/secret-type=repository
kubectl get secret -n argocd arcana-deployment-repo \
  -o go-template='{{index .metadata.labels "argocd.argoproj.io/secret-type"}}{{"\n"}}{{range $key, $value := .data}}{{printf "%s\n" $key}}{{end}}'

6. SOPS, age, and KSOPS

The age private key stays only on trusted operator machines:

export SOPS_AGE_KEY_FILE="$HOME/.config/sops/age/keys.txt"
test -r "$SOPS_AGE_KEY_FILE"
test "$(stat -c '%a' "$SOPS_AGE_KEY_FILE")" = 600
age-keygen -y "$SOPS_AGE_KEY_FILE"
rg -n 'age: age1' .sops.yaml

Validate encrypted input without exposing plaintext:

sops filestatus apps/arcana/overlays/production/backend-secret.sops.yaml
sops --decrypt apps/arcana/overlays/production/backend-secret.sops.yaml \
  | kubeconform -strict -summary -exit-on-error

Create the Argo CD age Secret:

kubectl create secret generic sops-age --namespace=argocd \
  --from-file=keys.txt="$SOPS_AGE_KEY_FILE"
kubectl get secret -n argocd sops-age \
  -o go-template='{{range $key, $value := .data}}{{printf "%s\n" $key}}{{end}}'

Configure the KSOPS repo-server sidecar and verify the rollout:

kubectl apply -f bootstrap/argocd/ksops/plugin-configmap.yaml
kubectl patch deployment argocd-repo-server --namespace=argocd \
  --type=strategic --patch-file=bootstrap/argocd/ksops/repo-server-patch.yaml \
  --dry-run=server -o name
kubectl rollout status -n argocd deployment/argocd-repo-server --timeout=5m
kubectl exec -n argocd deployment/argocd-repo-server -c ksops -- kustomize version

A KSOPS render test is more meaningful than running ksops with a conventional help flag.

7. Register Argo CD Applications

The Application must contain spec.project:

kubectl apply -f bootstrap/argocd/arcana-production.yaml
kubectl get application -n argocd arcana-production
kubectl get application -n argocd arcana-production \
  -o jsonpath='automated={.spec.syncPolicy.automated}{"\n"}enabled={.spec.syncPolicy.automated.enabled}{"\n"}'

Automated synchronization remains disabled during review. Argo can render and report drift but cannot create production workloads until a reviewed sync is requested:

kubectl annotate application -n argocd arcana-production \
  argocd.argoproj.io/refresh=hard --overwrite
kubectl get application -n argocd arcana-production \
  -o custom-columns='NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status,REVISION:.status.sync.revision'

8. Render and validate before applying

Kustomize confinement made system /tmp unreliable, so use a private cache:

RENDER_PARENT="$HOME/.cache/arcana-deployment-renders"
mkdir -p "$RENDER_PARENT"
chmod 700 "$RENDER_PARENT"
RENDER_TMP="$(mktemp -d "$RENDER_PARENT/gate-a.XXXXXX")"
mkdir -p "$RENDER_TMP/apps/arcana/overlays/production"
cp -R apps/arcana/base "$RENDER_TMP/apps/arcana/base"
sed '/^generators:/,$d' apps/arcana/overlays/production/kustomization.yaml \
  > "$RENDER_TMP/apps/arcana/overlays/production/kustomization.yaml"
kubectl kustomize "$RENDER_TMP/apps/arcana/overlays/production" \
  > "$RENDER_TMP/gate-a.yaml"
kubeconform -strict -summary -exit-on-error "$RENDER_TMP/gate-a.yaml"

Gate A checks schema, images, replica boundaries, and that no migration Job is registered:

if rg -n 'backend-migration-job\.yaml' apps/arcana/base/kustomization.yaml; then
  echo 'STOP: migration Job is registered too early' >&2
  exit 1
fi
git diff --check

Gate B temporarily renders the migration hook. It uses Argo Sync, a negative sync wave, and deletes the Job after success; regular application replicas stay at zero.

9. Inspect and execute the database migration

Check the image's Alembic head in the image itself:

BACKEND_IMAGE_TAG="$(awk '
  $1 == "-" && $2 == "name:" && $3 == "<BACKEND_IMAGE>" { in_backend=1; next }
  in_backend && $1 == "newTag:" { print $2; exit }
' apps/arcana/overlays/production/kustomization.yaml)"
printf '%s\n' "$BACKEND_IMAGE_TAG" | rg -x '[0-9a-f]{40}' >/dev/null
docker run --rm --platform linux/amd64 \
  --entrypoint /app/.venv/bin/alembic "<BACKEND_IMAGE>:$BACKEND_IMAGE_TAG" \
  -c /app/alembic.ini heads

Request a non-pruning operation at the reviewed revision:

REVISION="$(git rev-parse HEAD)"
kubectl patch application -n argocd arcana-production --type=merge \
  --patch "{\"operation\":{\"initiatedBy\":{\"username\":\"migration-guide\"},\"sync\":{\"revision\":\"$REVISION\",\"prune\":false}}}"
kubectl get application -n argocd arcana-production \
  -o jsonpath='{.status.operationState.phase}{" "}{.status.operationState.syncResult.revision}{"\n"}'
kubectl get jobs -n arcana

After the hook succeeds, remove it from the active Kustomization, render again, commit, and push. Retain the historical manifest if useful, but never leave a completed one-shot hook in active desired state.

10. Activate services one at a time

Start with replicas zero for backend, frontend, Celery worker, and beat; keep Redis available:

kubectl get deployment -n arcana \
  -o custom-columns='NAME:.metadata.name,DESIRED:.spec.replicas,READY:.status.readyReplicas'
kubectl get statefulset -n arcana
kubectl get pods -n arcana -o wide
kubectl get jobs -n arcana

Change one replica in Git, validate, commit, push, refresh, then request the reviewed sync. The backend runs directly with Uvicorn:

command: ["/app/.venv/bin/uvicorn"]
args: ["app:app", "--host", "0.0.0.0", "--port", "8000"]

Health checks use a separate port-forward terminal:

kubectl port-forward -n arcana service/arcana-backend 18000:8000

curl --proto '=http' -fsS http://127.0.0.1:18000/api/health/
curl --proto '=http' -fsS http://127.0.0.1:18000/api/health/db

The frontend is checked similarly. A 307 from / is an expected login redirect:

kubectl rollout status -n arcana deployment/arcana-frontend --timeout=5m
kubectl port-forward -n arcana service/arcana-frontend 13000:3000
curl --proto '=http' -fsS -o /dev/null \
  -w 'status=%{http_code} content_type=%{content_type}\n' \
  http://127.0.0.1:13000/

11. Activate Celery as nobody, not root

The worker command and queues are explicit:

command: ["/app/.venv/bin/celery"]
args: [-A, celery_app, worker, --loglevel=info, --queues=email,notifications,celery,dead_letter, --concurrency=1]

The image did not contain a named UID-1000 entry, which caused misleading Celery warnings. Use the existing unprivileged identity:

securityContext:
  runAsNonRoot: true
  runAsUser: 65534
  runAsGroup: 65534
  fsGroup: 65534
  fsGroupChangePolicy: OnRootMismatch

65534 is the conventional nobody/nogroup identity:

kubectl exec -n arcana deployment/arcana-celery-worker -- id
kubectl rollout status -n arcana deployment/arcana-celery-worker --timeout=5m
kubectl get pod -n arcana -l app.kubernetes.io/name=arcana-celery-worker \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .status.initContainerStatuses[*]}{.name}={.state.terminated.reason}{"\n"}{end}{end}'
kubectl exec -n arcana deployment/arcana-celery-worker -- \
  /app/.venv/bin/celery -A celery_app inspect ping --timeout=10
kubectl exec -n arcana deployment/arcana-celery-worker -- \
  /app/.venv/bin/celery -A celery_app inspect registered --timeout=10

Check metrics through a port-forward:

kubectl port-forward -n arcana service/arcana-celery-worker-metrics 18001:8001
curl --proto '=http' -fsS -o /dev/null \
  -w 'status=%{http_code} content_type=%{content_type}\n' \
  http://127.0.0.1:18001/metrics

Activate beat only after worker health is proven.

12. Move avatars from Docker to a PVC

The old stack shared a host bind mount at /avatar. Kubernetes uses a named claim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: arcana-backend-avatars
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 1Gi

Inventory the old source without following a link:

ssh vps '
  AVATAR_SOURCE="$(docker inspect tarot-backend \
    --format "{{range .Mounts}}{{if eq .Destination "/avatar"}}{{.Source}}{{end}}{{end}}")"
  test -n "$AVATAR_SOURCE"
  test -d "$AVATAR_SOURCE"
  test ! -L "$AVATAR_SOURCE"
  printf "source=%s\n" "$AVATAR_SOURCE"
  find "$AVATAR_SOURCE" -xdev -type f -printf . | wc -c
'

Copy content with no ownership restoration:

ssh vps 'tar -C <ABSOLUTE_AVATAR_SOURCE> -cpf - .' \
  | kubectl exec -i -n arcana deployment/arcana-backend -- \
      tar -C /avatar --no-same-owner -xpf -

Without --no-same-owner, tar attempted to restore a UID/GID that the unprivileged container could not set.

Verify and hash across a rollout:

kubectl get pvc -n arcana arcana-backend-avatars -o wide
kubectl exec -n arcana deployment/arcana-backend -- \
  /bin/sh -ec 'test -d /avatar; touch /avatar/.write-test; rm /avatar/.write-test'
kubectl exec -n arcana deployment/arcana-backend -- \
  tar -C /avatar -cf - . | sha256sum
kubectl rollout restart -n arcana deployment/arcana-backend
kubectl rollout status -n arcana deployment/arcana-backend --timeout=5m
kubectl exec -n arcana deployment/arcana-backend -- \
  tar -C /avatar -cf - . | sha256sum

The digests must match.

13. Retire Docker only after Kubernetes is healthy

ssh vps 'docker ps --filter name=tarot-backend --filter name=tarot-frontend \
  --filter name=tarot-celery-worker --filter name=tarot-celery-beat \
  --filter name=tarot-redis --filter name=traefik \
  --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"'
kubectl get pods -n arcana -o wide
kubectl top pods -n arcana
kubectl top node myvps
ssh vps 'df -h /; sudo du -sh /var/lib/rancher'

Stop only explicitly identified old Compose services after the Kubernetes checks pass. Never use a broad docker system prune on a shared VPS. Verify edge ownership:

ssh vps 'sudo ss -lntup | grep -E ":(80|443|6443)\b"'

14. Immutable GitOps promotion

GitHub Actions tests the application, builds and pushes backend/frontend images tagged by source SHA, and updates arcana-deployment. Argo CD consumes only that deployment repository. The obsolete imperative deployment workflow was removed so two systems cannot race.

images:
  - name: <BACKEND_IMAGE>
    newTag: <40-CHARACTER-GIT-SHA>
  - name: <FRONTEND_IMAGE>
    newTag: <40-CHARACTER-GIT-SHA>

Verify images and commit only intended files:

docker manifest inspect <OWNER>/tarot-backend:<GIT_SHA> >/dev/null
docker manifest inspect <OWNER>/tarot-frontend:<GIT_SHA> >/dev/null
git diff --check
git status --short
git add <specific-files>
git commit -m "<focused change>"
git push origin main

Refresh and verify the exact observed revision:

kubectl annotate application -n argocd arcana-production \
  argocd.argoproj.io/refresh=hard --overwrite
kubectl get application -n argocd arcana-production \
  -o jsonpath='revision={.status.sync.revision}{"\n"}sync={.status.sync.status}{"\n"}health={.status.health.status}{"\n"}'

15. Cloudflare-protected Argo CD UI

In Cloudflare Zero Trust:

  1. Create or reuse the remotely managed tunnel named argocd.
  2. Add public hostname argocd.<DOMAIN>.
  3. Route it to HTTPS service argocd-server.argocd.svc.cluster.local:443.
  4. Enable No TLS Verify for the internal Argo CD origin.
  5. Create an Access application for the hostname.
  6. Add an administrator allow policy.
  7. Test login in the browser.

The tunnel uses no public Kubernetes service and no new inbound VPS port. Its token is encrypted:

SECRET_PATH=apps/infrastructure/cloudflared-argocd/token-secret.sops.yaml
sops filestatus "$SECRET_PATH"
sops --decrypt "$SECRET_PATH" | kubeconform -strict -summary -exit-on-error
rg -F 'token: ENC[' "$SECRET_PATH"

The connector uses two replicas and a pinned image. QUIC/UDP 7844 failed on this VPS while TCP/HTTP2 passed, so the final arguments force HTTP2:

replicas: 2
containers:
  - name: cloudflared
    image: cloudflare/cloudflared:<PINNED_VERSION>
    args:
      - tunnel
      - --no-autoupdate
      - --metrics
      - 0.0.0.0:2000
      - --protocol
      - http2
      - run

Verify:

kubectl rollout status -n cloudflare deployment/cloudflared-argocd --timeout=5m
kubectl get pods -n cloudflare -o wide
kubectl logs -n cloudflare deployment/cloudflared-argocd --tail=100 \
  | rg 'Initial protocol|Registered tunnel connection|ERR|WRN'
kubectl get application -n argocd cloudflared-argocd \
  -o jsonpath='sync={.status.sync.status}{"\n"}health={.status.health.status}{"\n"}'

Healthy logs show Initial protocol http2 and registered connections.

16. Final checklist and rollback

export KUBECONFIG="$HOME/.kube/arcana-k3s.yaml"
kubectl get application -n argocd arcana-production \
  -o jsonpath='revision={.status.sync.revision}{"\n"}sync={.status.sync.status}{"\n"}health={.status.health.status}{"\n"}enabled={.spec.syncPolicy.automated.enabled}{"\n"}operation={.operation}{"\n"}'
kubectl get deployment -n arcana \
  -o custom-columns='NAME:.metadata.name,DESIRED:.spec.replicas,READY:.status.readyReplicas,AVAILABLE:.status.availableReplicas'
kubectl get statefulset -n arcana
kubectl get pods -n arcana -o wide
kubectl get pvc -n arcana
kubectl get jobs -n arcana
kubectl get ingress -n arcana
kubectl top pods -n arcana
kubectl top node myvps

Confirm the desired revision, sync, health, replicas, PVCs, absence of unexpected Jobs, and Cloudflare Access protection. For rollback, revert the deployment repository commit, push it, refresh Argo CD, and request a reviewed sync at that revision. Do not edit a live Deployment and leave Git divergent.

17. Troubleshooting lessons

  • macOS stat uses -f; Ubuntu stat uses -c.
  • A malformed kubeconfig is a YAML problem, not a K3s outage.
  • localhost:8080 means KUBECONFIG was lost.
  • A confined Kustomize binary may not see system /tmp; use $HOME/.cache.
  • Argo Applications require spec.project.
  • Missing sops, kubeconform, yq, or age-keygen is a local tooling problem.
  • sops --extract and Go templates inspect metadata without printing Secret values.
  • tar --no-same-owner avoids ownership failures.
  • Frontend 307 at / is expected login behavior.
  • UID 65534 is nobody/nogroup and avoids root.
  • Cloudflare HTTP2 was selected because QUIC UDP 7844 timed out.
  • Port-forwards must remain open while curls run.

The result is a delivery system that builds once, records the exact artifact in Git, decrypts secrets only where needed, runs migrations explicitly, activates workloads in stages, preserves uploads, and exposes Argo CD through an authenticated tunnel instead of an open Kubernetes service.