{"schemaVersion":"1.0","type":"TechArticle","types":["Article","TechArticle"],"slug":"kubernetes-etcd-backup-and-restore-1ciiy","url":"https://zyvop.com/kubernetes-etcd-backup-and-restore-1ciiy","title":"Kubernetes ETCD Backup And Restore","subtitle":null,"tldr":"A beginner-friendly, hands-on walkthrough of backing up and restoring etcd in a kubeadm-based Kubernetes cluster — including the mistakes, errors, debugging ste...","keywords":[],"entities":["Adeoye Malumi","ZyVOP"],"keyTakeaways":["A beginner-friendly, hands-on walkthrough of backing up and restoring etcd in a kubeadm-based Kubernetes cluster — including the mistakes, errors, debugging steps, and the terminal commands that actually happened during the lab.","Introduction If you are learning Kubernetes, you will eventually hear a simple but important statement: etcd is the database of a Kubernetes cluster.","That description is short, but it has a major consequence: if the etcd data is lost or corrupted, the Kubernetes control plane can lose the information it needs to know what exists in the cluster."],"headings":["Introduction","1. First: What is etcd?","Terminal","Lesson 1: The filename was wrong","Lesson 2: The command was deprecated","Endpoint","CA certificate","Client certificate","Private key","Snapshot command","Error 1: Wrong directory","Fix","Error 2: Docker container not found","Lesson","Error 3: Permission denied reading etcd files","Error 4: Invalid etcdctl flag","Error 5: Private key permission denied","Error 6: Wrong snapshot filename","Error 7: etcdctl snapshot status deprecated"],"outboundLinks":[],"contentText":"A beginner-friendly, hands-on walkthrough of backing up and restoring etcd in a kubeadm-based Kubernetes cluster — including the mistakes, errors, debugging steps, and the terminal commands that actually happened during the lab. Introduction If you are learning Kubernetes, you will eventually hear a simple but important statement: etcd is the database of a Kubernetes cluster. That description is short, but it has a major consequence: if the etcd data is lost or corrupted, the Kubernetes control plane can lose the information it needs to know what exists in the cluster. In this lab, I practiced an end-to-end etcd backup and restore workflow on a single-node kubeadm cluster running on a machine named osboxes. The exercise covered: identifying the etcd pod and confirming that the control plane was healthy; locating the static etcd manifest; inspecting the etcd configuration; installing etcdctl and etcdutl; creating an etcd snapshot; validating the snapshot; deliberately creating an nginx Deployment so there was something meaningful to restore; deleting the Deployment; restoring the etcd snapshot into a separate data directory; changing the static etcd manifest to use the restored data; troubleshooting an etcd pod that remained Pending; restarting kubelet; confirming that etcd eventually returned to Running. The important part of this exercise was not simply getting a successful backup command. It was understanding what each error meant and how to reason from the evidence in the terminal. The terminal session showed the etcd pod running before the backup, with the control plane components also healthy. Later, after changing the etcd static pod configuration, etcd became Pending; the investigation eventually led to restarting kubelet, after which etcd became 1/1 Running. The original terminal output is the source for the commands and results reproduced throughout this article. 1. First: What is etcd? Before touching commands, it helps to understand what we are backing up. Kubernetes has many components, but the important relationship for this exercise is roughly: kubectl | v kube-apiserver | v etcd | +---- Kubernetes cluster state The API server is the main interface through which Kubernetes clients interact with the cluster. etcd is the persistent key-value store used by Kubernetes to store cluster state. That state includes information such as: Kubernetes objects; Deployments; Pods and their specifications; Services; ConfigMaps; Secrets; cluster configuration and metadata. A useful mental model for a beginner is: The Kubernetes API is how you ask Kubernetes about the cluster; etcd is where much of that cluster state is persisted. This is why backing up etcd is fundamentally different from simply exporting Kubernetes YAML. 2. Verify that the cluster is healthy first Before taking a backup, I first checked the cluster. Terminal ┌──(osboxes㉿osboxes)-[~] └─$ k get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-74c68c8864-fjtnv 0/1 ContainerCreating 0 3m10s kube-system calico-node-dzlcp 0/1 Init:2/3 0 3m10s kube-system coredns-589f44dc88-l522w 0/1 ContainerCreating 0 8m56s kube-system coredns-589f44dc88-nzsh9 0/1 ContainerCreating 0 8m56s kube-system etcd-osboxes 1/1 Running 0 9m6s kube-system kube-apiserver-osboxes 1/1 Running 0 9m9s kube-system kube-controller-manager-osboxes 1/1 Running 0 9m5s kube-system kube-proxy-d8krd 1/1 Running 0 8m56s kube-system kube-scheduler-osboxes 1/1 Running 0 9m8s The most important observation here is: etcd-osboxes 1/1 Running The API server, controller manager, scheduler and kube-proxy were also running. Some networking components were still starting, so I watched the pods: ┌──(osboxes㉿osboxes)-[~] └─$ k get pods -A -w Eventually CoreDNS and Calico became ready. I then checked the node: ┌──(osboxes㉿osboxes)-[~] └─$ k get nodes NAME STATUS ROLES AGE VERSION osboxes Ready control-plane 10m v1.36.3 At this point the single control-plane node was Ready. The initial cluster state is important because a backup should be taken from a functioning etcd instance rather than blindly assuming that etcd is healthy. 3. An important distinction: Kubernetes YAML backup vs etcd backup Before working directly with etcd, I also ran: ┌──(osboxes㉿osboxes)-[~] └─$ k get all -A -o yaml > backup.yaml This produced: ┌──(osboxes㉿osboxes)-[~] └─$ ls backup.yaml Documents kubectl Pictures Public Videos Desktop Downloads Music Projects Templates VM-Share This is useful, but it is not the same thing as an etcd snapshot. The command: k get all -A -o yaml > backup.yaml exports Kubernetes resources returned by kubectl get all. An etcd snapshot, on the other hand, is a snapshot of the etcd backend itself. For an etcd disaster-recovery exercise, the etcd snapshot is the important artifact. 4. Finding the etcd static pod This cluster was created with kubeadm, so etcd was running as a static pod. I initially made a typo: ┌──(osboxes㉿osboxes)-[~] └─$ cd /etc/kkubenetes/manifests bash: cd: /etc/kkubenetes/manifests: No such file or directory The problem was simply the spelling: kkubenetes instead of: kubernetes I then made another typo: ┌──(osboxes㉿osboxes)-[~] └─$ cd /etc/kubenetes/manifests bash: cd: /etc/kubenetes/manifests: No such file or directory Again, the correct directory is: /etc/kubernetes/manifests Eventually: ┌──(osboxes㉿osboxes)-[~] └─$ cd /etc/kubernetes/manifests and: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ ls etcd.yaml kube-apiserver.yaml kube-controller-manager.yaml kube-scheduler.yaml This directory is extremely important on a kubeadm control-plane node. The files in /etc/kubernetes/manifests are static pod manifests. The kubelet watches this directory and makes sure the described control-plane containers are running. That means editing etcd.yaml is effectively changing how the control-plane etcd process is launched. 5. Another wrong assumption: trying to use Docker I also tried: ┌──(osboxes㉿osboxes)-[~] └─$ docker exec -it osboxes bash Error response from daemon: No such container: osboxes This failed because the Kubernetes node name osboxes is not necessarily a Docker container name. In this setup, the node was running Kubernetes with containerd, and the etcd pod later showed a container ID beginning with: containerd:// So the important lesson is: Do not assume that a Kubernetes node or Pod name is a Docker container name. If you need to inspect a Kubernetes container, first determine which container runtime the node is using. 6. Inspecting the etcd manifest The first attempt to read the manifest without elevated privileges also failed: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ cat etcd.yaml cat: etcd.yaml: Permission denied The solution was: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ sudo cat etcd.yaml The manifest revealed several important values. For example: - --advertise-client-urls=https://10.0.2.15:2379 - --cert-file=/etc/kubernetes/pki/etcd/server.crt - --client-cert-auth=true - --data-dir=/var/lib/etcd - --key-file=/etc/kubernetes/pki/etcd/server.key - --listen-client-urls=https://127.0.0.1:2379,https://10.0.2.15:2379 It also showed that the etcd data directory was originally: /var/lib/etcd and the TLS certificates were under: /etc/kubernetes/pki/etcd/ This information is essential because etcdctl needs to connect to the correct endpoint and authenticate with the appropriate certificates. 7. Install etcdctl The machine did not initially have the etcd client installed, so I installed it: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ sudo apt install etcd-client The package installed successfully: Installing: etcd-client Summary: Upgrading: 0, Installing: 1, Removing: 0, Not Upgrading: 205 I then checked the snapshot command: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ ETCDCTL_API=3 etcdctl snapshot The output showed: COMMANDS: restore Restores an etcd member snapshot to an etcd directory save Stores an etcd node backend snapshot to a given file status [deprecated] Gets backend snapshot status of a given file The key command for the backup is: etcdctl snapshot save 8. ETCDCTL_API=3 I also explicitly exported: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ export ETCDCTL_API=3 For the lab, this made it explicit that etcdctl should use the v3 API. Then: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ etcdctl snapshot again showed the snapshot subcommands. For a beginner, the important idea is: ETCDCTL_API=3 selects the etcd API version used by the client. 9. Creating the first etcd snapshot — and making a typo I attempted the backup with: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ etcdctl ---endpoint=https://127.0.0.1:2379,https://10.0.2.15:2379 \\ > --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ > --cert=/etc/kubernetes/pki/etcd/server.crt \\ > --key=/etc/kubernetes/pki/etcd/server.key \\ > snapshot save /opt/etcd-backup.db The error was: Error: bad flag syntax: ---endpoint=https://127.0.0.1:2379,https://10.0.2.15:2379 The mistake was the number of hyphens. I typed: ---endpoint instead of: --endpoints There were actually two problems worth noticing: There were three hyphens. The correct flag is --endpoints, plural. This is a classic CLI problem: the command can look almost correct to a human while being completely invalid to the parser. 10. The next error: permission denied I corrected the flag and tried: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ etcdctl --endpoints=https://127.0.0.1:2379 \\ --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt \\ --key=/etc/kubernetes/pki/etcd/server.key \\ snapshot save /opt/etcd-backup.db The command reached the certificate files but failed with: Error: open /etc/kubernetes/pki/etcd/server.key: permission denied This made sense. The etcd private key is protected, and the regular user did not have permission to read it. The solution was to run the command with sudo: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ sudo etcdctl --endpoints=https://127.0.0.1:2379 \\ --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt \\ --key=/etc/kubernetes/pki/etcd/server.key \\ snapshot save /opt/etcd-backup.db This time the backup succeeded. 11. The successful etcd backup The successful command produced: {\"level\":\"info\",\"ts\":\"2026-08-13T12:11:59.649233-0400\",\"caller\":\"snapshot/v3_snapshot.go:65\",\"msg\":\"created temporary db file\",\"path\":\"/opt/etcd-backup.db.part\"} {\"level\":\"info\",\"ts\":\"2026-08-13T12:11:59.660384-0400\",\"logger\":\"client\",\"caller\":\"v3/maintenance.go:212\",\"msg\":\"opened snapshot stream; downloading\"} {\"level\":\"info\",\"ts\":\"2026-08-13T12:11:59.660467-0400\",\"caller\":\"snapshot/v3_snapshot.go:73\",\"msg\":\"fetching snapshot\",\"endpoint\":\"https://127.0.0.1:2379\"} {\"level\":\"info\",\"ts\":\"2026-08-13T12:11:59.831074-0400\",\"logger\":\"client\",\"caller\":\"v3/maintenance.go:220\",\"msg\":\"completed snapshot read; closing\"} {\"level\":\"info\",\"ts\":\"2026-08-13T12:11:59.894020-0400\",\"caller\":\"snapshot/v3_snapshot.go:88\",\"msg\":\"fetched snapshot\",\"endpoint\":\"https://127.0.0.1:2379\",\"size\":\"3.9 MB\",\"took\":\"now\"} {\"level\":\"info\",\"ts\":\"2026-08-13T12:11:59.894158-0400\",\"caller\":\"snapshot/v3_snapshot.go:97\",\"msg\":\"saved\",\"path\":\"/opt/etcd-backup.db\"} Snapshot saved at /opt/etcd-backup.db The resulting backup file was: /opt/etcd-backup.db The snapshot was approximately: 3.9 MB This was the first major milestone. 12. Validating the snapshot I initially tried: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ etcdctl --write-out=table snapshot status snapshot.db The client responded: Deprecated: Use `etcdutl snapshot status` instead. Error: stat snapshot.db: no such file or directory There were two lessons here. Lesson 1: The filename was wrong The backup was saved as: /opt/etcd-backup.db not: snapshot.db Lesson 2: The command was deprecated The output explicitly recommended: etcdutl snapshot status So I attempted: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ etcdutl --write-out=table snapshot status snapshot.db but etcdutl was not installed. The system suggested: sudo apt install etcd-server I installed the package. After installation, I still used the wrong filename once: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ etcdutl --write-out=table snapshot status snapshot.db Error: stat snapshot.db: no such file or directory Then I used the correct path: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ etcdutl --write-out=table snapshot status /opt/etcd-backup.db This produced: Error: open /opt/etcd-backup.db: permission denied Again, the issue was file permissions. So I used sudo: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db The result was: ┌──────────┬──────────┬────────────┬────────────┐ │ HASH │ REVISION │ TOTAL KEYS │ TOTAL SIZE │ ├──────────┼──────────┼────────────┼────────────┤ │ a633855e │ 5973 │ 1298 │ 3.9 MB │ └──────────┴──────────┴────────────┴────────────┘ This confirmed that the snapshot was readable and contained etcd data. 13. Why create an nginx Deployment? A backup is much more meaningful if we have something we can verify after restoring it. At this point, I created an nginx Deployment. My first attempt was: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k create deploy nginx error: required flag(s) \"image\" not set The command was missing the container image. I then tried: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k create deploy --image=nginx error: exactly one NAME is required, got 0 See 'kubectl create deployment -h' for help and examples Now the image was present, but the Deployment name was missing. The correct command was: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k create deploy nginx --image=nginx deployment.apps/nginx created This is a good beginner lesson: kubectl create deployment <NAME> --image=<IMAGE> So: kubectl create deployment nginx --image=nginx means: create a Deployment; call it nginx; use the nginx container image. 14. Waiting for the nginx Pod Immediately after creation: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k get all NAME READY STATUS RESTARTS AGE pod/nginx-7f8fbb96d-9f7wm 0/1 ContainerCreating 0 4s The Pod remained in: ContainerCreating I watched it: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k get po -w After waiting, I checked again. The Pod eventually became: nginx-7f8fbb96d-9f7wm 1/1 Running 0 3m16s The describe output explained why it initially took time: Events: Type Reason Age From Message ---- ------ --- ---- ------- Normal Scheduled 83s default-scheduler Successfully assigned default/nginx-7f8fbb96d-9f7wm to osboxes Normal Pulling 82s kubelet spec.containers{nginx}: Pulling image \"nginx\" The Pod had been scheduled successfully. The kubelet was simply pulling the image. This is an important troubleshooting principle: ContainerCreating does not automatically mean something is broken. Check the Pod events before changing anything. Eventually: k get po NAME READY STATUS RESTARTS AGE nginx-7f8fbb96d-9f7wm 1/1 Running 0 3m16s 15. Take another backup after the nginx Deployment exists This step is important because the first snapshot was taken before the nginx Deployment was created. After nginx was running, I created another snapshot: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ sudo etcdctl --endpoints=https://127.0.0.1:2379 \\ --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt \\ --key=/etc/kubernetes/pki/etcd/server.key \\ snapshot save /opt/etcd-backup.db The snapshot was again: 3.9 MB and the command ended with: Snapshot saved at /opt/etcd-backup.db I validated it: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db Result: ┌──────────┬──────────┬────────────┬────────────┐ │ HASH │ REVISION │ TOTAL KEYS │ TOTAL SIZE │ ├──────────┼──────────┼────────────┼────────────┤ │ eb688bf4 │ 6934 │ 918 │ 3.9 MB │ └──────────┴──────────┴────────────┴────────────┘ Notice that the revision changed. The backup now represented a later state of the cluster. 16. Verify the state we want to recover At this point: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k get all NAME READY STATUS RESTARTS AGE pod/nginx-7f8fbb96d-9f7wm 1/1 Running 0 4m1s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 76m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/nginx 1/1 1 1 4m2s NAME DESIRED CURRENT READY AGE replicaset.apps/nginx-7f8fbb96d 1 1 1 4m2s The important object is: deployment.apps/nginx It was healthy. Now the test was simple: If I delete nginx, can I use the etcd snapshot to recover the cluster state that existed when the snapshot was taken? 17. Delete nginx I deleted the Deployment: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k delete deploy nginx deployment.apps \"nginx\" deleted from default namespace Then: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k get all NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 76m The nginx Deployment was gone. This gave us the desired recovery scenario: BEFORE BACKUP | +-- nginx Deployment exists | v TAKE ETCD SNAPSHOT | v DELETE nginx | v nginx no longer exists | v RESTORE SNAPSHOT | v expect previous cluster state 18. First restore mistake: wrong snapshot filename I attempted: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ etcdutl --data-dir /opt/etcd/backup snapshot restore snapshot.db The restore process immediately told me: Error: open snapshot.db: no such file or directory Again, the filename was wrong. The actual snapshot was: /opt/etcd-backup.db This is a very common command-line mistake: creating a file with one name and later trying to restore a similarly named file. The first restore attempt was therefore not an etcd failure. It was simply a path/filename error. 19. Restore the snapshot into a separate data directory Instead of overwriting the active etcd data directory immediately, I restored the snapshot into a separate directory: /var/lib/etcd-restore-from-backup The command used was: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ sudo etcdctl \\ --endpoints=https://127.0.0.1:2379 \\ --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt \\ --key=/etc/kubernetes/pki/etcd/server.key \\ snapshot restore /opt/etcd-backup.db \\ --data-dir=/var/lib/etcd-restore-from-backup The output included: Deprecated: Use `etcdutl snapshot restore` instead. and then: restoring snapshot followed by: Trimming membership information from the backend... and: added member Finally: restored snapshot This was the actual successful restore of the snapshot into the new data directory. 20. Why use a separate restore directory? The original etcd data directory was: /var/lib/etcd The restored copy was: /var/lib/etcd-restore-from-backup This distinction is important. It allows the restored data to be prepared without immediately destroying the existing data directory. For a beginner, think of it like restoring a database backup into a new folder first: Original: /var/lib/etcd Restored: /var/lib/etcd-restore-from-backup Only after the restored data exists do we configure etcd to use it. 21. More path mistakes while inspecting the restored data I attempted: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ cd /var/lib/data bash: cd: /var/lib/data: No such file or directory Again, the directory did not exist. The restore directory was: /var/lib/etcd-restore-from-backup I also tried: sudo cd /var/lib/etcd-restore-from-backup and got: sudo: cd: command not found sudo: \"cd\" is a shell built-in command, it cannot be run directly. This error is worth understanding. cd is a shell built-in command. sudo normally executes an external command, so: sudo cd ... does not work. Instead, because the directory required elevated permissions, I used: sudo ls /var/lib/etcd-restore-from-backup which showed: member That confirmed that the restore had created the expected etcd member directory structure. 22. Editing the static etcd manifest The next step was to configure the static etcd pod to use the restored data directory. The original manifest used: --data-dir=/var/lib/etcd The modified configuration used: --data-dir=/var/lib/etcd-restore-from-backup The corresponding hostPath was also changed to: /var/lib/etcd-restore-from-backup The resulting Pod description later confirmed: --data-dir=/var/lib/etcd-restore-from-backup and: Path: /var/lib/etcd-restore-from-backup This is how the restored database was connected to the etcd process. 23. Why moving the manifest files caused disruption During the troubleshooting process, I temporarily moved the static pod manifests out of the manifest directory: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ sudo mv * /tmp Then: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ ls -lrt total 0 The directory was empty. I then restored the manifests: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ sudo mv /tmp/*.yaml . and confirmed: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ ls -lrt total 16 -rw------- 1 root root 3939 Aug 13 11:07 kube-apiserver.yaml -rw------- 1 root root 3230 Aug 13 11:07 kube-controller-manager.yaml -rw------- 1 root root 1726 Aug 13 11:07 kube-scheduler.yaml -rw------- 1 root root 2596 Aug 13 12:37 etcd.yaml This demonstrates how sensitive the static pod manifest directory is. When the kubelet stops seeing a static pod manifest, the corresponding static pod can disappear. For this reason, moving files in /etc/kubernetes/manifests should be done carefully. 24. Another small typo I accidentally ran: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k get oi error: the server doesn't have a resource type \"oi\" The intended command was clearly not oi. The correct command for Pods was: k get po This is a harmless error, but it illustrates another important habit: When kubectl reports that a resource type does not exist, first check the command spelling before assuming the Kubernetes API is broken. 25. The RBAC error after the manifest changes After the static pod manipulation, this happened: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k get po Error from server (Forbidden): pods is forbidden: User \"kubernetes-admin\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\" This was different from the earlier errors. This time, the Kubernetes API server was responding, but the authenticated user was being denied access. The command: sudo k get po was also not a solution: sudo: k: command not found The reason is that k was a shell alias, and aliases normally are not available to sudo in that way. More importantly, sudo does not automatically solve Kubernetes RBAC problems. I then checked the system namespace explicitly: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k get po -n=kube-system This returned the control-plane Pods. That helped separate the problem from the actual etcd state. 26. etcd was now Pending The output showed: etcd-osboxes 0/1 Pending I first tried: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k describe po etcd-osboxes Error from server (NotFound): pods \"etcd-osboxes\" not found Again, the namespace mattered. The correct command was: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k describe po -n=kube-system etcd-osboxes The description showed the important configuration: Status: Pending but also: Image: registry.k8s.io/etcd:3.6.8-0 and: --data-dir=/var/lib/etcd-restore-from-backup The Pod also showed the restored directory mounted: /var/lib/etcd-restore-from-backup from etcd-data and: Path: /var/lib/etcd-restore-from-backup The kubelet events showed: Normal Pulled ... Normal Created ... Normal Started ... So the container had actually been created and started even though the Kubernetes status remained Pending. This is an important troubleshooting clue: Do not rely on one status field alone. Read the full Pod description and events. 27. Trying to read etcd logs I tried: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k logs etcd-osboxes -n kube-system Error from server (BadRequest): container \"etcd\" in pod \"etcd-osboxes\" is not available I also accidentally ran: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k logs which returned: error: expected 'logs [-f] [-p] (POD | TYPE/NAME) [-c CONTAINER]'. POD or TYPE/NAME is a required argument for the logs command The second error was simply because kubectl logs requires a Pod or workload name. The first error was more interesting: Kubernetes did not consider the container available for normal log retrieval. I then checked the Pod in wide format: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k get pod etcd-osboxes -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES etcd-osboxes 0/1 Pending 0 4m58s <none> osboxes <none> <none> The lack of a Pod IP was another indication that the Pod was not fully ready. 28. Inspecting kubelet Since etcd was a static Pod, the kubelet was a key component to inspect. I ran: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ sudo systemctl status kubelet --no-pager The result showed: ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/usr/lib/systemd/system/kubelet.service; disabled; preset: disabled) Active: active (running) The kubelet was running. The logs also showed repeated activity around the time of the problem. At this point, rather than making more changes to the etcd manifest, I restarted kubelet: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ sudo systemctl restart kubelet Then: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ systemctl daemon-reload 29. The recovery After restarting kubelet, I checked the system namespace again: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k get po -n=kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-74c68c8864-fjtnv 1/1 Running 5 (7m47s ago) 95m calico-node-dzlcp 1/1 Running 0 95m coredns-589f44dc88-l522w 1/1 Running 0 101m coredns-589f44dc88-nzsh9 1/1 Running 0 101m etcd-osboxes 1/1 Running 0 6m51s kube-apiserver-osboxes 1/1 Running 0 101m kube-controller-manager-osboxes 1/1 Running 1 (7m32s ago) 101m kube-proxy-d8krd 1/1 Running 0 101m kube-scheduler-osboxes 1/1 Running 1 101m This was the desired state. Most importantly: etcd-osboxes 1/1 Running 30. Confirming the restored etcd configuration I then described the etcd Pod again: ┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests] └─$ k describe po -n=kube-system etcd-osboxes The important portion was: Status: Running IP: 10.0.2.15 The etcd container was: State: Running Ready: True Restart Count: 0 And the command still showed: --data-dir=/var/lib/etcd-restore-from-backup The mounted volume was: /var/lib/etcd-restore-from-backup from etcd-data with the host path: /var/lib/etcd-restore-from-backup This confirmed that etcd was now running using the restored data directory. 31. What happened to the nginx Deployment? The exercise was designed around the snapshot that was taken while nginx existed. The important sequence was: 1. Create nginx Deployment 2. Wait for nginx to become Running 3. Take etcd snapshot 4. Delete nginx 5. Restore the snapshot 6. Point etcd at restored data 7. Restart kubelet 8. Confirm etcd is Running The backup validation showed a snapshot revision of: 6934 with: 918 total keys 3.9 MB total size The restore operation successfully rebuilt an etcd data directory. The final terminal state confirmed that etcd was successfully running from: /var/lib/etcd-restore-from-backup The terminal session provided here does not contain a final k get all after the successful etcd recovery showing nginx reappearing, so I would not claim that nginx restoration was independently verified from the terminal evidence. That distinction matters. The exercise successfully demonstrated: snapshot creation; snapshot validation; snapshot restoration; switching the static etcd Pod to restored data; recovery of the etcd Pod. The source terminal output does not provide a final explicit post-recovery k get all proving that the nginx Deployment was recreated from the restored snapshot. 32. The complete clean backup command For this environment, the working backup command was: sudo etcdctl \\ --endpoints=https://127.0.0.1:2379 \\ --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt \\ --key=/etc/kubernetes/pki/etcd/server.key \\ snapshot save /opt/etcd-backup.db The important pieces are: Endpoint --endpoints=https://127.0.0.1:2379 This tells etcdctl where etcd is listening for client connections. CA certificate --cacert=/etc/kubernetes/pki/etcd/ca.crt This allows the client to verify the etcd server certificate. Client certificate --cert=/etc/kubernetes/pki/etcd/server.crt This identifies the client to the TLS-enabled etcd server. Private key --key=/etc/kubernetes/pki/etcd/server.key This goes with the client certificate. Snapshot command snapshot save /opt/etcd-backup.db This tells etcdctl to create a backend snapshot at the specified path. 33. The clean snapshot validation command The session showed that etcdctl snapshot status is deprecated and recommended etcdutl. The working validation command was: sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db Example output from the session: ┌──────────┬──────────┬────────────┬────────────┐ │ HASH │ REVISION │ TOTAL KEYS │ TOTAL SIZE │ ├──────────┼──────────┼────────────┼────────────┤ │ eb688bf4 │ 6934 │ 918 │ 3.9 MB │ └──────────┴──────────┴────────────┴────────────┘ This is useful because a backup file existing on disk does not, by itself, prove that it is a valid etcd snapshot. 34. The clean restore command The restore performed successfully with: sudo etcdctl \\ --endpoints=https://127.0.0.1:2379 \\ --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt \\ --key=/etc/kubernetes/pki/etcd/server.key \\ snapshot restore /opt/etcd-backup.db \\ --data-dir=/var/lib/etcd-restore-from-backup The terminal explicitly reported: Deprecated: Use `etcdutl snapshot restore` instead. So for a newer workflow, prefer the modern etcdutl snapshot restore command rather than copying the deprecated command blindly. 35. Important troubleshooting lessons Error 1: Wrong directory cd /etc/kkubenetes/manifests and: cd /etc/kubenetes/manifests failed because of spelling mistakes. Fix cd /etc/kubernetes/manifests Error 2: Docker container not found docker exec -it osboxes bash returned: No such container: osboxes Lesson A Kubernetes node name is not automatically a Docker container name. Error 3: Permission denied reading etcd files cat etcd.yaml returned: Permission denied Fix sudo cat etcd.yaml The same principle applied to the protected etcd private key. Error 4: Invalid etcdctl flag ---endpoint=... returned: Error: bad flag syntax Fix Use the correctly spelled flag: --endpoints=... Error 5: Private key permission denied Error: open /etc/kubernetes/pki/etcd/server.key: permission denied Fix Run the etcd client command with sufficient privileges: sudo etcdctl ... Error 6: Wrong snapshot filename snapshot.db did not exist. The actual file was: /opt/etcd-backup.db Lesson Always verify the exact backup path before restoring. Error 7: etcdctl snapshot status deprecated The client reported: Deprecated: Use `etcdutl snapshot status` instead. Fix Install/use etcdutl and run: sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db Error 8: etcdutl not installed The command: etcdutl was initially unavailable. Fix The system suggested installing etcd-server, which provided the utility in this environment. Error 9: sudo cd This failed: sudo cd /var/lib/etcd-restore-from-backup with: sudo: cd: command not found Lesson cd is a shell built-in. Use commands such as: sudo ls /var/lib/etcd-restore-from-backup or start a privileged shell if you genuinely need one. Error 10: Pod name not found This: k describe po etcd-osboxes returned: pods \"etcd-osboxes\" not found because the command defaulted to the default namespace. Fix k describe po -n=kube-system etcd-osboxes Error 11: kubectl logs without a Pod k logs returned: POD or TYPE/NAME is a required argument for the logs command Fix Provide the resource: k logs <pod-name> and specify the namespace when needed. Error 12: kubectl Forbidden After the control-plane changes: Error from server (Forbidden): pods is forbidden: User \"kubernetes-admin\" cannot list resource \"pods\" This was an authorization/RBAC problem rather than a shell permission problem. sudo did not solve it because Kubernetes authorization is separate from Linux sudo. Error 13: etcd remained Pending The etcd Pod stayed: 0/1 Pending even though the container had been created and started according to the events. The investigation included: k describe po -n=kube-system etcd-osboxes and: sudo systemctl status kubelet --no-pager The practical recovery step that resolved the state in this lab was: sudo systemctl restart kubelet After that: etcd-osboxes 1/1 Running 36. Why static Pods matter in this exercise The etcd Pod was not created with a normal Deployment such as: kind: Deployment Instead, it was defined by: /etc/kubernetes/manifests/etcd.yaml That makes it a static Pod. The kubelet watches the manifest directory and manages the Pod locally. This explains several things that happened during the exercise: changing etcd.yaml affected etcd; temporarily moving manifests out of the directory affected the control plane; restarting kubelet caused it to re-read and reconcile the manifests; etcd could be controlled without creating a normal Kubernetes Deployment. For kubeadm clusters, understanding static Pods is essential when troubleshooting the control plane. 37. A safer mental model for etcd restore Do not think of restore as: \"copy backup file over etcd\" A better mental model is: ETCD SNAPSHOT | v restore snapshot | v new etcd data directory | v configure etcd to use restored directory | v kubelet recreates/restarts static etcd Pod | v etcd becomes healthy | v Kubernetes API becomes available against restored state This is why the exercise used: /var/lib/etcd-restore-from-backup instead of immediately destroying: /var/lib/etcd 38. What I would do differently next time The lab worked, but it also exposed several areas where the workflow could be cleaner. 1. Check the exact paths first Before typing a long command: ls -l /etc/kubernetes/pki/etcd/ ls -l /opt/etcd-backup.db This avoids guessing filenames and permissions. 2. Inspect the manifest before changing it First: sudo cp /etc/kubernetes/manifests/etcd.yaml /root/etcd.yaml.backup Then edit the manifest. That gives you a known-good copy if something goes wrong. 3. Prefer the current etcd utilities The terminal explicitly showed that: etcdctl snapshot status is deprecated and recommended: etcdutl snapshot status Likewise, the restore output recommended: etcdutl snapshot restore 4. Avoid unnecessary movement of all manifests This command: sudo mv * /tmp is powerful and potentially disruptive. Moving all control-plane manifests affects more than etcd. A safer approach is to modify only the required manifest and keep backups of the original. 5. Verify each stage After every major step: k get pods -A or: k get pods -n kube-system and inspect: k describe pod <pod> -n kube-system if anything looks abnormal. 39. A compact recovery checklist For a kubeadm single-control-plane lab similar to this one: Identify etcd k get pods -n kube-system | grep etcd Inspect the manifest cd /etc/kubernetes/manifests sudo cat etcd.yaml Identify: --data-dir --listen-client-urls --cert-file --key-file --trusted-ca-file Create snapshot sudo etcdctl \\ --endpoints=https://127.0.0.1:2379 \\ --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt \\ --key=/etc/kubernetes/pki/etcd/server.key \\ snapshot save /opt/etcd-backup.db Validate snapshot sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db Restore to a separate directory sudo etcdutl snapshot restore /opt/etcd-backup.db \\ --data-dir=/var/lib/etcd-restore-from-backup Configure etcd to use the restored directory Update: --data-dir=/var/lib/etcd-restore-from-backup and the corresponding hostPath. Restart/reconcile kubelet if required sudo systemctl restart kubelet Verify k get pods -n kube-system Look for: etcd-osboxes 1/1 Running 40. What this lab actually taught me The most valuable part of this exercise was not memorizing one command. It was learning how the pieces connect: Kubernetes | +-- kube-apiserver | +-- kube-controller-manager | +-- kube-scheduler | +-- kubelet | +-- static Pod manifests | +-- etcd | +-- /var/lib/etcd The backup process interacted directly with etcd: etcd | +-- TLS certificates | +-- etcdctl | +-- snapshot | +-- /opt/etcd-backup.db The restore process then created: /var/lib/etcd-restore-from-backup and the static Pod was changed to consume that restored data. Finally, kubelet reconciled the static Pod and etcd returned to: 1/1 Running 41. Final takeaway An etcd backup is not just another Kubernetes YAML export. The core workflow is: 1. Verify etcd is healthy. 2. Find the etcd endpoint and TLS certificates. 3. Take an etcd snapshot. 4. Validate the snapshot. 5. Restore the snapshot into a separate data directory. 6. Configure the etcd static Pod to use the restored data. 7. Let kubelet reconcile the static Pod. 8. Verify etcd and the Kubernetes control plane. The terminal session also demonstrated something that documentation often hides: the commands do not always work on the first attempt. There were: spelling mistakes; incorrect flags; permission problems; wrong filenames; deprecated commands; missing utilities; namespace mistakes; RBAC errors; attempts to use sudo where it did not apply; a static etcd Pod that remained Pending; and finally a kubelet restart that brought etcd back to Running. Those failures were not separate from the learning experience. They were the learning experience. The most useful troubleshooting habit from this lab is simple: When Kubernetes gives you an error, do not immediately change random things. Read the error, identify which layer produced it, inspect the relevant component, make the smallest reasonable correction, and verify the result. That is the difference between memorizing Kubernetes commands and actually learning how the control plane works. Terminal command reference from the lab Cluster health k get pods -A k get pods -A -w k get nodes k get all k get all -A Static Pod manifests cd /etc/kubernetes/manifests ls sudo cat etcd.yaml Install tooling sudo apt install etcd-client sudo apt install etcd-server Configure etcdctl export ETCDCTL_API=3 Backup sudo etcdctl \\ --endpoints=https://127.0.0.1:2379 \\ --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt \\ --key=/etc/kubernetes/pki/etcd/server.key \\ snapshot save /opt/etcd-backup.db Validate sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db Test workload k create deploy nginx --image=nginx k get po k get po -w k describe po nginx-7f8fbb96d-9f7wm k get all Delete workload k delete deploy nginx Restore sudo etcdctl \\ --endpoints=https://127.0.0.1:2379 \\ --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt \\ --key=/etc/kubernetes/pki/etcd/server.key \\ snapshot restore /opt/etcd-backup.db \\ --data-dir=/var/lib/etcd-restore-from-backup Troubleshoot etcd k get po -n=kube-system k describe po -n=kube-system etcd-osboxes k get pod etcd-osboxes -n kube-system -o wide k logs etcd-osboxes -n kube-system sudo systemctl status kubelet --no-pager sudo systemctl restart kubelet Final verification k get po -n=kube-system k describe po -n=kube-system etcd-osboxes","contentHash":"sha256:f186c7e8616d5be6c37e8361c05d711bbc9b5229e11178a27a0b638296413ccb","authorName":"Adeoye Malumi","authorUrl":"https://zyvop.com/author/adeoye","authorSameAs":[],"category":null,"tags":[],"audience":"Senior software engineers, systems architects, and technical leads working with software development","tone":"Instructional, practical, code-first","readingTimeMinutes":23,"wordCount":5906,"faqs":null,"primaryTopic":null,"publishedAt":"2026-08-18T09:01:04.645Z","updatedAt":"2026-08-22T04:40:00.242Z","canonicalUrl":"https://zyvop.com/kubernetes-etcd-backup-and-restore-1ciiy"}