My Notes

Setup Rook Ceph RBD Snapshots Hourly

5 min read

Setup Rook Ceph RBD Snapshots on Hourly Schedule in Kubernetes

This guide walks through the process of configuring Rook Ceph RBD (RADOS Block Device) volume snapshots on an hourly schedule in a Kubernetes cluster.

Prerequisites

  • A working Kubernetes cluster (v1.20+)
  • Rook Ceph cluster already deployed and operational
  • kubectl access to your cluster
  • Storage class based on Ceph RBD provisioner
  • CSI snapshot controller and CRDs installed

Step 1: Verify CSI Snapshot Support

First, ensure the CSI snapshot controller and required CRDs are installed:

# Check if VolumeSnapshotClass CRD exists
kubectl get crd volumesnapshotclasses.snapshot.storage.k8s.io
 
# Check if snapshot controller is running (usually in kube-system namespace)
kubectl get pods -n kube-system | grep snapshot-controller

If these are not installed, you’ll need to install them:

# Install snapshot CRDs
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-6.3/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-6.3/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-6.3/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
 
# Install snapshot controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-6.3/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-6.3/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml

Step 2: Create a VolumeSnapshotClass

Create a VolumeSnapshotClass that will be used to define the snapshot properties:

# rbd-snapshot-class.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ceph-rbd-snapshot-class
driver: rook-ceph.rbd.csi.ceph.com
parameters:
  # Specify parameters for your Ceph cluster if needed
  clusterID: rook-ceph
  csi.storage.k8s.io/snapshotter-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/snapshotter-secret-namespace: rook-ceph
deletionPolicy: Delete  # or Retain based on your requirements

Apply this configuration:

kubectl apply -f rbd-snapshot-class.yaml

Step 3: Create a CronJob for Scheduled Snapshots

Now we’ll create a CronJob that runs every hour to create snapshots of your RBD volumes.

First, create a ServiceAccount with proper permissions:

# snapshot-creator-rbac.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: snapshot-creator
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: snapshot-creator
rules:
- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["get", "list"]
- apiGroups: ["snapshot.storage.k8s.io"]
  resources: ["volumesnapshots"]
  verbs: ["create", "get", "list", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: snapshot-creator
subjects:
- kind: ServiceAccount
  name: snapshot-creator
  namespace: default
roleRef:
  kind: ClusterRole
  name: snapshot-creator
  apiGroup: rbac.authorization.k8s.io

Apply the RBAC configuration:

kubectl apply -f snapshot-creator-rbac.yaml

Next, create the CronJob that will create snapshots:

# hourly-snapshot-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: hourly-volume-snapshots
  namespace: default
spec:
  schedule: "0 * * * *"  # Run every hour at minute 0
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: snapshot-creator
          containers:
          - name: snapshot-creator
            image: bitnami/kubectl:latest
            command:
            - /bin/bash
            - -c
            - |
              # Get today's date in YYYY-MM-DD-HHMM format
              DATE=$(date +%Y-%m-%d-%H%M)
              
              # Get all PVCs that use our RBD storage class
              # Replace rook-ceph-block with your actual storage class name
              PVCS=$(kubectl get pvc -o jsonpath='{range .items[?(@.spec.storageClassName=="rook-ceph-block")]}{.metadata.name}{"\n"}{end}')
              
              # Create a snapshot for each PVC
              for PVC in $PVCS; do
                # Check if PVC is bound
                STATUS=$(kubectl get pvc $PVC -o jsonpath='{.status.phase}')
                if [ "$STATUS" == "Bound" ]; then
                  echo "Creating snapshot for PVC $PVC"
                  cat <<EOF | kubectl apply -f -
                  apiVersion: snapshot.storage.k8s.io/v1
                  kind: VolumeSnapshot
                  metadata:
                    name: ${PVC}-snapshot-${DATE}
                    namespace: default
                  spec:
                    volumeSnapshotClassName: ceph-rbd-snapshot-class
                    source:
                      persistentVolumeClaimName: ${PVC}
              EOF
                else
                  echo "Skipping PVC $PVC as it's not in Bound state"
                fi
              done
              
              # Optional: Cleanup old snapshots (keep last 24 hourly snapshots)
              # Uncomment to enable cleanup
              # OLD_SNAPSHOTS=$(kubectl get volumesnapshot --sort-by=.metadata.creationTimestamp -o jsonpath='{range .items[0:-24]}{.metadata.name}{"\n"}{end}')
              # for SNAPSHOT in $OLD_SNAPSHOTS; do
              #   echo "Deleting old snapshot $SNAPSHOT"
              #   kubectl delete volumesnapshot $SNAPSHOT
              # done
          restartPolicy: OnFailure

Apply the CronJob:

kubectl apply -f hourly-snapshot-cronjob.yaml

Step 4: Verify and Monitor Snapshot Creation

To verify that snapshots are being created correctly:

# Check if the CronJob exists
kubectl get cronjob hourly-volume-snapshots
 
# Check the most recent job execution
kubectl get job
 
# List all volume snapshots
kubectl get volumesnapshot
 
# Describe a specific snapshot
kubectl describe volumesnapshot <snapshot-name>

Step 5: Configure Snapshot Retention Policy (Optional)

By default, the script is set to keep all snapshots. To enable automatic cleanup of old snapshots, uncomment the cleanup section in the CronJob YAML. This will keep only the most recent 24 hourly snapshots and delete older ones.

Step 6: Restoring from a Snapshot (When Needed)

To restore data from a snapshot, create a new PVC from the snapshot:

# restore-from-snapshot.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-restored-from-snapshot
spec:
  storageClassName: rook-ceph-block
  dataSource:
    name: pvc-name-snapshot-yyyy-mm-dd-hhmm
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi  # Must be >= size of original PVC

Apply the restoration PVC:

kubectl apply -f restore-from-snapshot.yaml

Troubleshooting

Common Issues:

  1. Snapshot creation fails:

    • Check if CSI snapshot controller is running
    • Verify the VolumeSnapshotClass parameters are correct
    • Check Ceph health status: kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status

  2. CronJob not running:

    • Check CronJob events: kubectl describe cronjob hourly-volume-snapshots
    • Check job logs: kubectl logs job/<job-name>

  3. Permission errors:

    • Verify the ServiceAccount has correct RBAC permissions
    • Check if the necessary secrets exist in the correct namespace

Advanced Configurations

Configure Snapshot Scheduling for Specific Namespaces

To create snapshots only for PVCs in specific namespaces, modify the CronJob script to filter PVCs by namespace:

NAMESPACE="your-namespace"
PVCS=$(kubectl get pvc -n $NAMESPACE -o jsonpath='{range .items[?(@.spec.storageClassName=="rook-ceph-block")]}{.metadata.name}{"\n"}{end}')

Custom Snapshot Naming and Labels

To add custom labels or annotations to snapshots, modify the snapshot YAML template in the CronJob:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: ${PVC}-snapshot-${DATE}
  namespace: default
  labels:
    app: your-app
    backup-type: hourly
  annotations:
    description: "Hourly automated backup"
spec:
  volumeSnapshotClassName: ceph-rbd-snapshot-class
  source:
    persistentVolumeClaimName: ${PVC}

This completes the setup of hourly Rook Ceph RBD snapshots on your Kubernetes cluster.

Graph View