🗄️ Shared Storage (RWX) with CephFS

This guide shows how to install Ceph with the Rook operator on a Kubernetes Cluster on Nuvion by SaveinCloud and create a CephFS, a shared file system.

With CephFS, the cluster provides ReadWriteMany (RWX) volumes: several pods, on different workers, mount and write to the same volume at the same time. This is useful for applications with several replicas that need the same files, such as website uploads, static content and configuration files.

The steps follow the official Rook documentation:

• Rook Ceph: Quickstart
• Rook Ceph: Shared Filesystem (CephFS)

---

🧩 1. How It Works

Component	What it does	Where it runs
Rook operator	Installs and manages Ceph inside Kubernetes	1 pod on a worker
Monitors (mon)	Keep the Ceph cluster map; need a majority (quorum)	3, one on each worker
Managers (mgr)	Ceph metrics and modules	2 (1 active, 1 standby)
OSDs	Store the data, one per disk	1 per worker, on the extra volume
MDS	CephFS metadata server	2 (1 active, 1 standby-replay)
CSI driver	Creates volumes and mounts them in pods	Plugin on each worker

CephFS data is stored with 3 replicas, each on a different worker. If a worker goes down, the data is still available on the other two.

---

⚠️ 2. Prerequisites

• 3 workers (recommended): Ceph uses 3 monitors and stores each piece of data on 3 different workers. With fewer than 3 workers, the cluster does not reach HEALTH_OK and the data has no redundancy.
• Worker resources: we recommend at least 4 vCPU and 8 GiB of RAM on each worker. With 1 vCPU and 2 GiB, the Ceph pods do not fit.
• One extra volume per worker: see the next step.
• Tools: kubectl with access to the cluster (see Cluster Management) and git.

---

💽 3. Creating the Volumes for the Workers

Ceph needs dedicated disks. Each worker must get a new, empty, unformatted volume, used only by Ceph.

Do not use the volumes the node already has

Each worker in the cluster already comes with two volumes, which must not be used by Ceph:

• Node volume (vda): the operating system disk.
• Container volume (vdb): where container images and data are stored, mounted at /var/lib/containerd.

Ceph must use a third volume (vdc), created only for it. This way, Ceph data does not compete for space with the system or with container images.

1. In the Nuvion panel, create one volume for each worker, all the same size. Since each piece of data is stored 3 times, the usable CephFS space is roughly the size of one volume. For example, 3 volumes of 10 GiB give about 10 GiB of usable space. See how to create a volume in Block Storage.

2. Attach each volume to a different worker (node-0, node-1 and node-2).

Do not format or mount the volumes

Rook only uses empty disks: no partition and no file system. Do not run mkfs or mount on these volumes. If the disk has a file system, Rook skips the disk and the OSD is not created.

3. Check that the volume shows up empty on each worker. Replace <worker-name> with the name shown by kubectl get nodes:

kubectl debug node/<worker-name> -it --image=busybox:1.36 --profile=sysadmin -- chroot /host lsblk -f

The new volume should show up as vdc, with nothing in the FSTYPE and MOUNTPOINTS columns:

NAME   FSTYPE  ...  MOUNTPOINTS
vda                                        ← node volume (system)
├─vda3 ext4         /boot
└─vda4 xfs          /sysroot ...
vdb    xfs          /var/lib/containerd    ← container volume
vdc                                        ← new volume, for Ceph

The command creates a node-debugger-... pod in the default namespace. When you are done, delete that pod with kubectl delete pod <pod-name> (the name is shown by kubectl get pods).

---

📥 4. Downloading the Rook Manifests

Clone the Rook repository at the desired version. This guide uses v1.21.0:

git clone --single-branch --branch v1.21.0 https://github.com/rook/rook.git
cd rook/deploy/examples

All files used from here on are in this directory.

---

⚙️ 5. Installing the Rook Operator

Create the CRDs, the common permissions, the CSI operator and the Rook operator:

kubectl create -f crds.yaml -f common.yaml -f csi-operator.yaml
kubectl create -f operator.yaml

Wait for the operator pod to be Running:

kubectl -n rook-ceph get pod -l app=rook-ceph-operator -w

---

🛠️ 6. Adjusting and Creating the Ceph Cluster

The example cluster.yaml is already set up for this scenario:

• useAllNodes: true and useAllDevices: true: Rook looks for empty disks on all nodes and creates an OSD on each one. It finds vdc and ignores vda and vdb, because those already have a file system.
• security.cephx.csi.keyType: aes: keys compatible with the nodes' kernel. Ceph 20 generates AES-256 keys by default, and the Fedora CoreOS 6.12 kernel does not accept that type; the mount would fail with Unknown error 524. Do not change this option.

Since Ceph 20 considers the aes type insecure, the cluster stays in HEALTH_WARN with insecure key types warnings. These warnings are expected in this scenario. To mute them, edit cluster.yaml:

vi cluster.yaml

Find the healthCheck: section and add the muteHealthWarning block:

  healthCheck:
    muteHealthWarning:
      AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE:
        policy: mute
      AUTH_INSECURE_CLIENT_KEY_TYPE:
        policy: mute
      AUTH_INSECURE_KEYS_ALLOWED:
        policy: mute
      AUTH_INSECURE_KEYS_CREATABLE:
        policy: mute
    daemonHealth:
      # ... (keep the rest as it is)

Create the cluster:

kubectl create -f cluster.yaml

tip

If you already created the cluster before editing the file, apply the change with kubectl apply -f cluster.yaml.

Creation takes a few minutes. Watch the pods until the mon, mgr and one osd per worker show up:

kubectl -n rook-ceph get pod -w

---

🧰 7. Installing the Toolbox and Checking Ceph

The toolbox is a pod with the ceph command line, used to check the cluster health:

kubectl create -f toolbox.yaml
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status

Expected result:

  cluster:
    health: HEALTH_OK
            (muted: AUTH_INSECURE_CLIENT_KEY_TYPE AUTH_INSECURE_KEYS_ALLOWED ...)

  services:
    mon: 3 daemons, quorum a,b,c
    mgr: b(active), standbys: a
    osd: 3 osds: 3 up, 3 in

Check that each worker has an OSD and that the OSDs are on the new volumes:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd tree
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd metadata | grep -E '"(hostname|devices)"'

Each OSD should show up on a different worker, with "devices": "vdc".

---

📂 8. Creating the File System (CephFS)

Create the myfs CephFS, with metadata and data replicated 3 times:

kubectl create -f filesystem.yaml

Wait for the MDS pods to be Running and check the file system:

kubectl -n rook-ceph get pod -l app=rook-ceph-mds
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph fs status myfs

You should see 1 active MDS and 1 standby-replay, on different workers.

---

🏷️ 9. Creating the StorageClass

Create the rook-cephfs StorageClass, which the PVCs will use:

kubectl apply -f csi/cephfs/storageclass.yaml
kubectl get sc

NAME          PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION
rook-cephfs   rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true

• allowVolumeExpansion: true: the volume can be expanded later.
• reclaimPolicy: Delete: when you delete the PVC, the volume data is deleted with it.

---

🧪 10. Testing the Shared Volume

To test, create an RWX PVC and a Deployment with 3 nginx replicas using the same volume. Then write a file from one pod and read it from the others.

1. Create the teste-pvc.yaml file:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: teste-pvc
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 2Gi
  storageClassName: rook-cephfs

kubectl apply -f teste-pvc.yaml
kubectl get pvc teste-pvc

The PVC should be Bound, with ACCESS MODES set to RWX.

2. Create the teste.yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-compartilhada
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-shared
  template:
    metadata:
      labels:
        app: web-shared
    spec:
      containers:
        - name: nginx
          image: nginx:alpine
          volumeMounts:
            - name: dados-compartilhados
              mountPath: /usr/share/nginx/html
      volumes:
        - name: dados-compartilhados
          persistentVolumeClaim:
            claimName: teste-pvc

kubectl apply -f teste.yaml
kubectl get pods -l app=web-shared -o wide -w

3. With the 3 pods Running, write a file from the first pod (replace <pod-1> with the real name):

kubectl exec -it <pod-1> -- sh -c "echo 'Hello from Pod 1 through CephFS' > /usr/share/nginx/html/index.html"

4. Read the file from the other pods:

kubectl exec -it <pod-2> -- cat /usr/share/nginx/html/index.html
kubectl exec -it <pod-3> -- cat /usr/share/nginx/html/index.html

All of them should show Hello from Pod 1 through CephFS. This confirms that the pods, even on different workers, are using the same volume.

5. When you are done, remove the test:

kubectl delete -f teste.yaml -f teste-pvc.yaml

---

📌 11. Usage Best Practices

Writing to the same file

Several pods can write to the same volume without problems when each one writes to its own files. When several pods write to the same file, the application must use locking (flock/fcntl), which CephFS honors across nodes. Without locking, simultaneous writes to the same file can lose data without any error showing up. NFS behaves the same way.

• Usable space: check free space with ceph df. Since data is stored 3 times, usable space is about 1/3 of the total size of the volumes.
• More space: attach one more empty volume to each worker. Rook creates a new OSD for each empty disk on the next reconciliation. To force the reconciliation right away, restart the operator: kubectl -n rook-ceph rollout restart deploy/rook-ceph-operator.
• Databases: for volumes used by a single pod, such as a database's, prefer a ReadWriteOnce volume. CephFS is meant for data that several pods need to access at the same time.

---

🩺 12. Troubleshooting

Symptom	Likely cause	Solution
No rook-ceph-osd-N pod shows up	The volume has a partition or file system, or was not attached	Check with lsblk -f (step 3). See the reason in the logs: kubectl -n rook-ceph logs -l app=rook-ceph-osd-prepare -c provision
HEALTH_WARN with undersized PGs	Fewer than 3 workers with an OSD	Make sure you have 3 workers, each with its own volume
HEALTH_WARN ... insecure key types	aes keys, required by the nodes' kernel	Mute the warnings with muteHealthWarning (step 6)
Pod stuck in ContainerCreating with Unknown error 524	AES-256 keys, which the nodes' kernel does not accept	Keep security.cephx.csi.keyType: aes in cluster.yaml
PVC stuck in Pending	The CephFS or the StorageClass was not created	Check with ceph fs status myfs and kubectl get sc rook-cephfs

---

🗑️ 13. Removal

danger

Removal deletes all Ceph data. Back up anything important before you continue.

Follow the order below, from the rook/deploy/examples directory:

# 1. Delete the workloads and PVCs that use rook-cephfs, then the StorageClass
kubectl get pvc -A | grep rook-cephfs
kubectl delete -f csi/cephfs/storageclass.yaml

# 2. Delete the CephFS
kubectl -n rook-ceph delete cephfilesystem myfs

# 3. Allow the data cleanup and delete the Ceph cluster
kubectl -n rook-ceph patch cephcluster rook-ceph --type merge \
  -p '{"spec":{"cleanupPolicy":{"confirmation":"yes-really-destroy-data"}}}'
kubectl -n rook-ceph delete cephcluster rook-ceph

# 4. Once the CephCluster is gone, remove the toolbox, the operator and the CRDs
kubectl -n rook-ceph get cephcluster
kubectl delete -f toolbox.yaml
kubectl delete -f operator.yaml
kubectl delete -f common.yaml -f csi-operator.yaml -f crds.yaml

Then detach and delete the worker volumes in the Nuvion panel. To reuse a volume in a new Ceph installation, it must be empty again, with no partition and no file system.

More details in Rook Ceph: Cleanup.