Velero (preview)

lionbackup is becoming a storage target for Velero, the backup tool for Kubernetes. For that there is a plugin that registers lionbackup as a BackupStorageLocation provider, and a small controller in the cluster that uploads every completed backup as one encrypted file into a lionbackup project. Both are available as beta 0.2.0-beta.1: for trying out in a test cluster, not yet as your only backup.

Preview: no volume data yet

What reaches lionbackup is the spool: Kubernetes manifests, metadata and logs of the backup. The contents of volumes (PVCs) are not backed up yet; a PVC comes back as a definition, but empty. Keep relying on another path for volume data.

What the preview does today, and what it does not

Works today	Still missing
Velero runs fully against a spool in the cluster: backup and restore of manifests, sync, GC, `backup delete`	Volume data: PVC contents are not captured (Velero's node-agent only knows s3, azure, gcs and filesystem)
Signed URLs: `velero backup logs` and `describe --details` work, backups end `Completed`	Restore import: the bundle returns to the spool from outside the cluster (client with a read token)
Upload: the controller bundles `backups/<name>/` into one `.lbk`, uploads it with a write token and annotates the backup with the `file_id`	Capture and restore jobs for volumes; retries only as "again in 10 minutes"

A Failed backup means nothing was written; usually the owner of the spool directory is wrong then (see step 1). A PartiallyFailed only occurs when the controller or the shared URL secret is missing (step 4).

Trying it out

You need a test cluster, kubectl, the Velero CLI (tested with Velero 1.18.3), a lionbackup project with a write token and the lionbackup client on your workstation for the key pair. The plugin image can be pulled without credentials.

1. Create the spool directory

Velero writes into a directory on the node that is mounted into the Velero pod as a hostPath (or as a PVC). The official Velero image runs as user ID 1002; fsGroup does not apply to a hostPath, so the directory has to be owned by that ID. Otherwise every backup fails with permission denied while still reporting all items as backed up; the error only shows in status.failureReason.

mkdir -p /var/lib/lionbackup/spool
chown -R 1002:1002 /var/lib/lionbackup/spool
chmod 775 /var/lib/lionbackup/spool

2. Install Velero with the plugin

velero install \
  --provider lionbackup.cloud/lionbackup \
  --plugins git.prod.lionbackup.cloud/lionbackup/velero-plugin-lionbackup:0.2.0-beta.1 \
  --bucket spool \
  --no-secret \
  --use-volume-snapshots=false \
  --backup-location-config spoolPath=/var/lib/lionbackup/spool \
  --wait

--no-secret is correct: the plugin itself needs no credentials. The write token belongs in the controller's Secret (step 4), never on the BackupStorageLocation.

3. Mount the spool into the Velero pod

kubectl -n velero patch deployment velero --type=json -p '[
 {"op":"add","path":"/spec/template/spec/volumes/-","value":{"name":"lionbackup-spool","hostPath":{"path":"/var/lib/lionbackup/spool","type":"DirectoryOrCreate"}}},
 {"op":"add","path":"/spec/template/spec/containers/0/volumeMounts/-","value":{"name":"lionbackup-spool","mountPath":"/var/lib/lionbackup/spool"}}
]'
kubectl -n velero rollout status deploy/velero --timeout=180s
kubectl -n velero get backupstoragelocation default

Without this step the spool lives in the pod's filesystem and is gone after the next restart. The BackupStorageLocation should report Available afterwards.

4. Controller and Secret

The controller runs the same image as user 1002, with the spool mounted read-only and the Secret at /etc/lionbackup. It needs three things: the project's write token, a random URL secret it shares with the Velero pod, and the public key for encryption. Generate the key pair on your workstation; the private key never enters the cluster, and without it there is no restore.

lionbackup --generate-key --key-name ./velero

kubectl -n velero create secret generic lionbackup-velero \
  --from-literal=token=<WRITE-TOKEN> \
  --from-literal=url-secret="$(head -c 32 /dev/urandom | base64)" \
  --from-file=key.pub=./velero.pub

Save the following manifest as controller.yaml. It contains ServiceAccount, Role, RoleBinding, Service and Deployment; the work directory /work has to hold the largest backup directory of the spool.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: lionbackup-velero-controller
  namespace: velero
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: lionbackup-velero-controller
  namespace: velero
rules:
  - apiGroups: ["velero.io"]
    resources: ["backups"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: ["velero.io"]
    resources: ["backupstoragelocations"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: lionbackup-velero-controller
  namespace: velero
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: lionbackup-velero-controller
subjects:
  - kind: ServiceAccount
    name: lionbackup-velero-controller
    namespace: velero
---
apiVersion: v1
kind: Service
metadata:
  name: lionbackup-velero-controller
  namespace: velero
spec:
  selector:
    app.kubernetes.io/name: lionbackup-velero-controller
  ports:
    - name: http
      port: 8080
      targetPort: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: lionbackup-velero-controller
  namespace: velero
  labels:
    app.kubernetes.io/name: lionbackup-velero-controller
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: lionbackup-velero-controller
  template:
    metadata:
      labels:
        app.kubernetes.io/name: lionbackup-velero-controller
    spec:
      serviceAccountName: lionbackup-velero-controller
      securityContext:
        runAsUser: 1002
        runAsGroup: 1002
        runAsNonRoot: true
      containers:
        - name: controller
          image: git.prod.lionbackup.cloud/lionbackup/velero-plugin-lionbackup:0.2.0-beta.1
          command: ["/plugins/velero-plugin-lionbackup"]
          args: ["controller"]
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: VELERO_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: LIONBACKUP_SPOOL_PATH
              value: /var/lib/lionbackup/spool
            - name: LIONBACKUP_KEYFILE
              value: /etc/lionbackup/key.pub
            - name: LIONBACKUP_WORKDIR
              value: /work
            - name: LIONBACKUP_TOKEN
              valueFrom:
                secretKeyRef:
                  name: lionbackup-velero
                  key: token
            - name: LIONBACKUP_VELERO_URL_SECRET
              valueFrom:
                secretKeyRef:
                  name: lionbackup-velero
                  key: url-secret
          readinessProbe:
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 3
          resources:
            requests:
              cpu: 50m
              memory: 128Mi
            limits:
              cpu: "2"
              memory: 1Gi
          volumeMounts:
            - name: lionbackup-spool
              mountPath: /var/lib/lionbackup/spool
              readOnly: true
            - name: lionbackup-secret
              mountPath: /etc/lionbackup
              readOnly: true
            - name: work
              mountPath: /work
      volumes:
        - name: lionbackup-spool
          hostPath:
            path: /var/lib/lionbackup/spool
            type: Directory
        - name: lionbackup-secret
          secret:
            secretName: lionbackup-velero
            items:
              - key: key.pub
                path: key.pub
        - name: work
          emptyDir:
            sizeLimit: 10Gi

Then apply it, hand the URL secret to the Velero deployment as well and put project and zone on the BackupStorageLocation:

kubectl apply -f controller.yaml
kubectl -n velero set env deployment/velero \
  LIONBACKUP_VELERO_URL_SECRET="$(kubectl -n velero get secret lionbackup-velero -o jsonpath='{.data.url-secret}' | base64 -d)"
kubectl -n velero patch backupstoragelocation default --type=merge -p \
  '{"spec":{"config":{"lbProject":"<PROJECT-UUID>","lbZone":"de01-1","lbEnvironment":"prod"}}}'
kubectl -n velero rollout status deploy/velero --timeout=180s
kubectl -n velero rollout status deploy/lionbackup-velero-controller --timeout=180s

Without token or key the controller only logs what it would upload (observe-only). Further config keys of the BackupStorageLocation: lbCompressionMethod (ZSTD), lbCompressionLevel (5), lbChunksizeMb (64), lbUploadRateLimitMbit (0 = unlimited).

The signed URLs point at the controller Service inside the cluster, so velero backup logs and describe --details work wherever that Service is reachable. If the Velero CLI runs outside the cluster, forward the port (kubectl -n velero port-forward svc/lionbackup-velero-controller 8080:8080) and set controllerURL on the BackupStorageLocation to http://localhost:8080; the signature covers path and expiry only, not the host.

5. Backup

velero backup create demo-1 --include-namespaces demo-app --wait
velero backup logs demo-1 | tail -n 3
kubectl -n velero get backup demo-1 \
  -o jsonpath='{.status.phase} {.metadata.annotations.lionbackup\.cloud/file-id}{"\n"}'

Expect Completed, and velero backup logs returns the log through the controller's signed URL. Shortly after, the Backup object carries the annotation lionbackup.cloud/file-id: the identifier of the uploaded file in the project, the same one the client's --list shows. If the upload fails, the reason is in the controller's log and it retries after ten minutes.

6. Restore from lionbackup

The way back starts outside the cluster, with a read token and the private key; both stay away from the cluster that way. The client fetches the bundle, you copy the backup directory into the target cluster's spool, and Velero picks it up on the next sync of the BackupStorageLocation:

lionbackup --config read.yaml --list
lionbackup --config read.yaml --restore <FILE-ID> --identity ./velero.key --target ./restored
# ./restored/…/backups/demo-1/ -> <spoolPath>/spool/backups/demo-1/ des Zielclusters
velero restore create demo-restore --from-backup demo-1 --wait

Manifests, deployments, ConfigMaps and PVC definitions come back. The data in the volumes does not; that is the documented limit of the preview.

What comes next

The next stage backs up volume contents: a job per PVC on the pod's node streams the volume into the spool as an archive, a matching restore job fills it back, and an init container holds the application until the volume is there again. On top comes a controller endpoint that pulls a bundle by file_id straight back into the spool. Until then the note at the top of this page applies.

What the preview does today, and what it does not​

Trying it out​

1. Create the spool directory​

2. Install Velero with the plugin​

3. Mount the spool into the Velero pod​

4. Controller and Secret​

5. Backup​

6. Restore from lionbackup​

What comes next​