How to Cancel or Undo Resource Deletion (with Finalizers) in Kubernetes?

The short answer to this question is that you cannot. For a slightly descriptive answer, keep reading.

Let’s say you try deleting a PersistentVolume (kubectl delete pv [pv_name]) that is bound to a PersistentVolumeClaim, it will get stuck in Terminating state.

$ k get pv
NAME      CAPACITY   RECLAIM POLICY   STATUS   CLAIM
pv0001    50Gi       Retain           Bound    test/myclaim

$ k get pvc
NAME      STATUS   VOLUME   CAPACITY
myclaim   Bound    pv0001   50Gi   

$ k delete pv pv0001
persistentvolume "pv0001" deleted

$ k get pv
NAME      CAPACITY   RECLAIM POLICY   STATUS        CLAIM
pv0001    50Gi       Retain           Terminating   test/myclaim

This is because the PV has a Finalizer (kubernetes.io/pv-protection) that prevents it from accidentally getting deleted when in use by a PVC:

$ k describe pv pv0001
Name:            pv0001
Finalizers:      [kubernetes.io/pv-protection]
...

This behaviour is also documented:

If a user deletes a PVC in active use by a Pod, the PVC is not removed immediately. PVC removal is postponed until the PVC is no longer actively used by any Pods. Also, if an admin deletes a PV that is bound to a PVC, the PV is not removed immediately. PV removal is postponed until the PV is no longer bound to a PVC.

Now the question is once a PV has been deleted (maybe accidentally) with kubectl delete pv [pv_name] and is in Terminating state, can we abort or rollback the deletion operation? The answer is NO.

Initially, I thought if I unset the metadata.deletionTimestamp on the PV (with kubectl edit), it would prevent the eventual deletion of the object but that is currently not allowed. On going through the GitHub issues of K8S, I discovered that the deletion of an object is irreversible. Once the metadata.deletionTimestamp is set, the object will be eventually deleted. The presence of a finalizer can delay the process indefinitely though.

When you DELETE an object, the object becomes read-only. It can disappear instantly if it has no finalizers but with finalizers, the metadata.deletionTimestamp gets set and finalizers can be removed (but not added or changed). Once the respective controllers remove all the finalizers of the object, it disappears and eventually gets purged from the cluster.

So if you delete a resource that you don’t want to eventually go away (even after the finalizers are removed), your only option is to make a copy of the resource and switch to using that.

Leave a Reply

Your email address will not be published. Required fields are marked *