Let’s cover everything you need to know about resizing Kubernetes volumes to expand their size (upsizing). Downsizing is not possible.

We will talk keeping persistent volumes in mind but most of what we discuss should also be applicable to Generic ephemeral volumes. Also when we say “resizing volumes”, we refer to resizing the actual storage medium (disks/volumes) along with the K8S resources (PV and PVC).

When provisioning volumes in K8S (dynamically or statically), the general setup is to have a StorageClass, a PersistentVolume and a PersistentVolumeClaim which is what a Pod uses to access the storage asset/backend (disks) via the PV.

Volume Expansion

Resizing volumes in K8S makes use of a feature called Volume Expansion which only works if the storage driver/volume plugin you are using supports it. For CSI (out-of-tree) volumes, you can check the support here whereas for in-tree volumes refer to this.

Also the PersistentVolumeClaimResize admission controller is enabled by default in the API server that prevents resizing of PVCs unless the relevant StorageClass has allowVolumeExpansion set to true. However, this does not prevent patching PVs directly to resize bounded PVC (we will see this below).

Patching PVC to Resize

Now to use Volume Expansion in order to bump up the storage capacity of the disk/volume, all we have to do is patch the PVC. This can be done by running kubectl edit pvc <pvc_name> and then changing the value for spec.resources.requests.storage. Another option is to run kubectl patch pvc ... to make the same change.

Let’s see an example where I will provision a volume dynamically using the GCP/GKE CSI Volume Plugin and then patch the PVC to resize it. Following are the relevant manifests:

# storageclass.yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard-custom
parameters:
  type: pd-standard
provisioner: pd.csi.storage.gke.io
reclaimPolicy: Delete
volumeBindingMode: Immediate
---
# pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  storageClassName: standard-custom
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 10Gi

Let’s create the SC and PVC now and inspect them:

$ k apply -f storageclass.yaml
storageclass.storage.k8s.io/standard-custom

$ k apply -f pvc.yaml
persistentvolumeclaim/myclaim

$ k get pvc
NAME      STATUS   VOLUME                                     CAPACITY   STORAGECLASS
myclaim   Bound    pvc-5826e99d-4cbd-48eb-bb20-47b45ffdbf23   10Gi       standard-custom

$ k get pv
NAME                                       CAPACITY   RECLAIM POLICY   STATUS   CLAIM          STORAGECLASS
pvc-5826e99d-4cbd-48eb-bb20-47b45ffdbf23   10Gi       Delete           Bound    test/myclaim   standard-custom

We’ve created a 10Gi PVC that has created a disk in GCP along with the PV object. Now let’s try to patch the PVC to resize it (via Volume Expansion):

$ k patch pvc myclaim -p '{ "spec": { "resources": { "requests": { "storage": "15Gi" }}}}'
persistentvolumeclaim/myclaim patched

# PVC Capacity is not updated yet!!!
# We will discuss this below, keep reading :)
$ k get pvc
NAME      STATUS   VOLUME                                     CAPACITY    STORAGECLASS
myclaim   Bound    pvc-5826e99d-4cbd-48eb-bb20-47b45ffdbf23   10Gi        standard-custom

# PV Capacity is updated successfully
$ k get pv
NAME                                       CAPACITY   RECLAIM POLICY   STATUS   CLAIM          STORAGECLASS
pvc-5826e99d-4cbd-48eb-bb20-47b45ffdbf23   15Gi       Delete           Bound    test/myclaim   standard-custom

Just a bit of additional information, when we run k describe pvc myclaim, we see events like these (related to re-sizing):

Events:
Type     Reason                    From                                      Message
----     ------                    ----                                      -------
Warning  ExternalExpanding         volume_expand                             Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.

Normal   Resizing                  external-resizer pd.csi.storage.gke.io    External resizer is resizing volume pvc-5826e99d-4cbd-48eb-bb20-47b45ffdbf23

Normal   FileSystemResizeRequired  external-resizer pd.csi.storage.gke.io    Require file system resize of volume on node

Now if you notice the PV capacity has been updated from 10Gi to 15Gi. That’s great but the PVC capacity is still the same (10Gi). This is because the PVC capacity will not be updated until the PV or the backing storage is actually mounted on a worker node and the file system on the disk is resized (by kubelet).

The solution to this is simple, we just need to create a Pod that references the volume, i.e., uses the PVC. Let’s create the following pod:

# pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pv-pod-1
spec:
  volumes:
    - name: pv-storage
      persistentVolumeClaim:
        claimName: myclaim
  containers:
    - name: web
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: pv-storage

Run the commands:

$ k apply -f pod.yaml
pod/pv-pod-1 created

# Now the PVC capacity is resized!!!
$ k get pvc
NAME      STATUS   VOLUME                                     CAPACITY   STORAGECLASS
myclaim   Bound    pvc-5826e99d-4cbd-48eb-bb20-47b45ffdbf23   15Gi       standard-custom

When the pod is created, the disk is mounted onto the worker node and its file system is resized by kubelet. If you describe the PVC again you will see an event for this too:

Type     Reason                        From       Message
----     ------                        ----       -------
Normal   FileSystemResizeSuccessful    kubelet    MountVolume.NodeExpandVolume succeeded for volume "pvc-5826e99d-4cbd-48eb-bb20-47b45ffdbf23" <node-name>

But what happens when a PVC is resized that is already in use by a running Pod? In that case, its capacity will be automatically updated, i.e., the file system expansion will happen immediately after the backing disk is resized by the volume plugin (pd.csi.storage.gke.io in the example above). This expansion is online and the pods don’t have to be taken down or restarted causing no application downtime.

So we have learnt how to resize K8S volumes by simply patching PVCs.

Patching PV to Resize

At this point, we might wonder what happens if instead of patching a PVC, we patch the relevant PV’s capacity. Let’s try it:

# Patch PV storage from 15Gi -> 20Gi
$ k patch pv pvc-4493aa21-6e6f-4195-a6ea-0bc51a3246d0 -p '{ "spec": { "capacity": { "storage": "20Gi" }}}'

If a pod is referencing the Volume, kubelet will try to expand the size and it’ll keep failing with the following event:

MountVolume.Setup failed while expanding volume for volume "pvc-4493aa21-6e6f-4195-a6ea-0bc51a3246d0" : Expander.NodeExpand failed to expand the volume : rpc error: code = Internal desc = resize requested for 21474836480 but after resize volume was size 16106127360

If you try to create a new pod with this Volume, it’ll get stuck in ContainerCreating state and never run successfully.

To fix this, we will have to manually expand the backing disk/volume in our cloud provider/on-prem. In my case, it is GCP so running the following command resizes the disk:

$ gcloud compute disks resize pvc-4493aa21-6e6f-4195-a6ea-0bc51a3246d0 --size=20GB --zone=us-west1-b

Once the disk is resized, kubelet will be able to resize the file system and allow the changes to reflect in an existing/new pod (can check with df -h). So before upsizing a PV resource directly, make sure you’ve actually resized the backing disk/volume first.

Also patching a PV currently circumvents the absence of allowVolumeExpansion: true in the StorageClass that doesn’t let patching PVCs to resize volumes (also mentioned in the Volume Expansion section above).