How to Cancel or Undo Resource Deletion (with Finalizers) in Kubernetes?
The short answer to this question is that you cannot. For a slightly descriptive answer, keep reading.
Let’s say you try deleting a PersistentVolume (kubectl delete pv [pv_name]
) that is bound to a PersistentVolumeClaim, it will get stuck in Terminating
state.
$ k get pv
NAME CAPACITY RECLAIM POLICY STATUS CLAIM
pv0001 50Gi Retain Bound test/myclaim
$ k get pvc
NAME STATUS VOLUME CAPACITY
myclaim Bound pv0001 50Gi
$ k delete pv pv0001
persistentvolume "pv0001" deleted
$ k get pv
NAME CAPACITY RECLAIM POLICY STATUS CLAIM
pv0001 50Gi Retain Terminating test/myclaim
This is because the PV has a Finalizer (kubernetes.io/pv-protection
) that prevents it from accidentally getting deleted when in use by a PVC:
$ k describe pv pv0001
Name: pv0001
Finalizers: [kubernetes.io/pv-protection]
...
This behaviour is also documented:
If a user deletes a PVC in active use by a Pod, the PVC is not removed immediately. PVC removal is postponed until the PVC is no longer actively used by any Pods. Also, if an admin deletes a PV that is bound to a PVC, the PV is not removed immediately. PV removal is postponed until the PV is no longer bound to a PVC.
Now the question is once a PV has been deleted (maybe accidentally) with kubectl delete pv [pv_name]
and is in Terminating
state, can we abort or rollback the deletion operation? The answer is NO.
Initially, I thought if I unset the metadata.deletionTimestamp
on the PV (with kubectl edit
), it would prevent the eventual deletion of the object but that is currently not allowed. On going through the GitHub issues of K8S, I discovered that the deletion of an object is irreversible. Once the metadata.deletionTimestamp
is set, the object will be eventually deleted. The presence of a finalizer can delay the process indefinitely though.
When you DELETE
an object, the object becomes read-only. It can disappear instantly if it has no finalizers but with finalizers, the metadata.deletionTimestamp
gets set and finalizers can be removed (but not added or changed). Once the respective controllers remove all the finalizers of the object, it disappears and eventually gets purged from the cluster.
So if you delete a resource that you don’t want to eventually go away (even after the finalizers are removed), your only option is to make a copy of the resource and switch to using that.