A beginner-friendly, hands-on walkthrough of backing up and restoring etcd in a kubeadm-based Kubernetes cluster — including the mistakes, errors, debugging steps, and the terminal commands that actually happened during the lab.

Introduction

If you are learning Kubernetes, you will eventually hear a simple but important statement:

etcd is the database of a Kubernetes cluster.

That description is short, but it has a major consequence: if the etcd data is lost or corrupted, the Kubernetes control plane can lose the information it needs to know what exists in the cluster.

In this lab, I practiced an end-to-end etcd backup and restore workflow on a single-node kubeadm cluster running on a machine named osboxes.

The exercise covered:

identifying the etcd pod and confirming that the control plane was healthy;
locating the static etcd manifest;
inspecting the etcd configuration;
installing etcdctl and etcdutl;
creating an etcd snapshot;
validating the snapshot;
deliberately creating an nginx Deployment so there was something meaningful to restore;
deleting the Deployment;
restoring the etcd snapshot into a separate data directory;
changing the static etcd manifest to use the restored data;
troubleshooting an etcd pod that remained Pending;
restarting kubelet;
confirming that etcd eventually returned to Running.

The important part of this exercise was not simply getting a successful backup command. It was understanding what each error meant and how to reason from the evidence in the terminal.

The terminal session showed the etcd pod running before the backup, with the control plane components also healthy. Later, after changing the etcd static pod configuration, etcd became Pending; the investigation eventually led to restarting kubelet, after which etcd became 1/1 Running. The original terminal output is the source for the commands and results reproduced throughout this article.

1. First: What is etcd?

Before touching commands, it helps to understand what we are backing up.

Kubernetes has many components, but the important relationship for this exercise is roughly:

kubectl
   |
   v
kube-apiserver
   |
   v
  etcd
   |
   +---- Kubernetes cluster state

The API server is the main interface through which Kubernetes clients interact with the cluster. etcd is the persistent key-value store used by Kubernetes to store cluster state.

That state includes information such as:

Kubernetes objects;
Deployments;
Pods and their specifications;
Services;
ConfigMaps;
Secrets;
cluster configuration and metadata.

A useful mental model for a beginner is:

The Kubernetes API is how you ask Kubernetes about the cluster; etcd is where much of that cluster state is persisted.

This is why backing up etcd is fundamentally different from simply exporting Kubernetes YAML.

2. Verify that the cluster is healthy first

Before taking a backup, I first checked the cluster.

Terminal

┌──(osboxes㉿osboxes)-[~]
└─$ k get pods -A
NAMESPACE     NAME                                       READY   STATUS              RESTARTS   AGE
kube-system   calico-kube-controllers-74c68c8864-fjtnv   0/1     ContainerCreating   0          3m10s
kube-system   calico-node-dzlcp                          0/1     Init:2/3             0          3m10s
kube-system   coredns-589f44dc88-l522w                   0/1     ContainerCreating   0          8m56s
kube-system   coredns-589f44dc88-nzsh9                   0/1     ContainerCreating   0          8m56s
kube-system   etcd-osboxes                               1/1     Running             0          9m6s
kube-system   kube-apiserver-osboxes                     1/1     Running             0          9m9s
kube-system   kube-controller-manager-osboxes            1/1     Running             0          9m5s
kube-system   kube-proxy-d8krd                           1/1     Running             0          8m56s
kube-system   kube-scheduler-osboxes                     1/1     Running             0          9m8s

The most important observation here is:

etcd-osboxes  1/1  Running

The API server, controller manager, scheduler and kube-proxy were also running.

Some networking components were still starting, so I watched the pods:

┌──(osboxes㉿osboxes)-[~]
└─$ k get pods -A -w

Eventually CoreDNS and Calico became ready.

I then checked the node:

┌──(osboxes㉿osboxes)-[~]
└─$ k get nodes
NAME      STATUS   ROLES           AGE   VERSION
osboxes   Ready    control-plane   10m   v1.36.3

At this point the single control-plane node was Ready.

The initial cluster state is important because a backup should be taken from a functioning etcd instance rather than blindly assuming that etcd is healthy.

3. An important distinction: Kubernetes YAML backup vs etcd backup

Before working directly with etcd, I also ran:

┌──(osboxes㉿osboxes)-[~]
└─$ k get all -A -o yaml > backup.yaml

This produced:

┌──(osboxes㉿osboxes)-[~]
└─$ ls
backup.yaml  Documents  kubectl  Pictures  Public  Videos
Desktop      Downloads  Music    Projects  Templates  VM-Share

This is useful, but it is not the same thing as an etcd snapshot.

The command:

k get all -A -o yaml > backup.yaml

exports Kubernetes resources returned by kubectl get all.

An etcd snapshot, on the other hand, is a snapshot of the etcd backend itself.

For an etcd disaster-recovery exercise, the etcd snapshot is the important artifact.

4. Finding the etcd static pod

This cluster was created with kubeadm, so etcd was running as a static pod.

I initially made a typo:

┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kkubenetes/manifests
bash: cd: /etc/kkubenetes/manifests: No such file or directory

The problem was simply the spelling:

kkubenetes

instead of:

kubernetes

I then made another typo:

┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kubenetes/manifests
bash: cd: /etc/kubenetes/manifests: No such file or directory

Again, the correct directory is:

/etc/kubernetes/manifests

Eventually:

┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kubernetes/manifests

and:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls
etcd.yaml
kube-apiserver.yaml
kube-controller-manager.yaml
kube-scheduler.yaml

This directory is extremely important on a kubeadm control-plane node.

The files in /etc/kubernetes/manifests are static pod manifests. The kubelet watches this directory and makes sure the described control-plane containers are running.

That means editing etcd.yaml is effectively changing how the control-plane etcd process is launched.

5. Another wrong assumption: trying to use Docker

I also tried:

┌──(osboxes㉿osboxes)-[~]
└─$ docker exec -it osboxes bash
Error response from daemon: No such container: osboxes

This failed because the Kubernetes node name osboxes is not necessarily a Docker container name.

In this setup, the node was running Kubernetes with containerd, and the etcd pod later showed a container ID beginning with:

containerd://

So the important lesson is:

Do not assume that a Kubernetes node or Pod name is a Docker container name.

If you need to inspect a Kubernetes container, first determine which container runtime the node is using.

6. Inspecting the etcd manifest

The first attempt to read the manifest without elevated privileges also failed:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ cat etcd.yaml
cat: etcd.yaml: Permission denied

The solution was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo cat etcd.yaml

The manifest revealed several important values.

For example:

- --advertise-client-urls=https://10.0.2.15:2379
- --cert-file=/etc/kubernetes/pki/etcd/server.crt
- --client-cert-auth=true
- --data-dir=/var/lib/etcd
- --key-file=/etc/kubernetes/pki/etcd/server.key
- --listen-client-urls=https://127.0.0.1:2379,https://10.0.2.15:2379

It also showed that the etcd data directory was originally:

/var/lib/etcd

and the TLS certificates were under:

/etc/kubernetes/pki/etcd/

This information is essential because etcdctl needs to connect to the correct endpoint and authenticate with the appropriate certificates.

7. Install etcdctl

The machine did not initially have the etcd client installed, so I installed it:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo apt install etcd-client

The package installed successfully:

Installing:
  etcd-client

Summary:
  Upgrading: 0, Installing: 1, Removing: 0, Not Upgrading: 205

I then checked the snapshot command:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ETCDCTL_API=3 etcdctl snapshot

The output showed:

COMMANDS:
    restore  Restores an etcd member snapshot to an etcd directory
    save     Stores an etcd node backend snapshot to a given file
    status   [deprecated] Gets backend snapshot status of a given file

The key command for the backup is:

etcdctl snapshot save

8. ETCDCTL_API=3

I also explicitly exported:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ export ETCDCTL_API=3

For the lab, this made it explicit that etcdctl should use the v3 API.

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl snapshot

again showed the snapshot subcommands.

For a beginner, the important idea is:

ETCDCTL_API=3

selects the etcd API version used by the client.

9. Creating the first etcd snapshot — and making a typo

I attempted the backup with:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl ---endpoint=https://127.0.0.1:2379,https://10.0.2.15:2379 \
> --cacert=/etc/kubernetes/pki/etcd/ca.crt \
> --cert=/etc/kubernetes/pki/etcd/server.crt \
> --key=/etc/kubernetes/pki/etcd/server.key \
> snapshot save /opt/etcd-backup.db

The error was:

Error: bad flag syntax: ---endpoint=https://127.0.0.1:2379,https://10.0.2.15:2379

The mistake was the number of hyphens.

I typed:

---endpoint

instead of:

--endpoints

There were actually two problems worth noticing:

There were three hyphens.
The correct flag is --endpoints, plural.

This is a classic CLI problem: the command can look almost correct to a human while being completely invalid to the parser.

10. The next error: permission denied

I corrected the flag and tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db

The command reached the certificate files but failed with:

Error: open /etc/kubernetes/pki/etcd/server.key: permission denied

This made sense.

The etcd private key is protected, and the regular user did not have permission to read it.

The solution was to run the command with sudo:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db

This time the backup succeeded.

11. The successful etcd backup

The successful command produced:

{"level":"info","ts":"2026-08-13T12:11:59.649233-0400","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/opt/etcd-backup.db.part"}
{"level":"info","ts":"2026-08-13T12:11:59.660384-0400","logger":"client","caller":"v3/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2026-08-13T12:11:59.660467-0400","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2026-08-13T12:11:59.831074-0400","logger":"client","caller":"v3/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2026-08-13T12:11:59.894020-0400","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"3.9 MB","took":"now"}
{"level":"info","ts":"2026-08-13T12:11:59.894158-0400","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/opt/etcd-backup.db"}
Snapshot saved at /opt/etcd-backup.db

The resulting backup file was:

/opt/etcd-backup.db

The snapshot was approximately:

3.9 MB

This was the first major milestone.

12. Validating the snapshot

I initially tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl --write-out=table snapshot status snapshot.db

The client responded:

Deprecated: Use `etcdutl snapshot status` instead.

Error: stat snapshot.db: no such file or directory

There were two lessons here.

Lesson 1: The filename was wrong

The backup was saved as:

/opt/etcd-backup.db

not:

snapshot.db

Lesson 2: The command was deprecated

The output explicitly recommended:

etcdutl snapshot status

So I attempted:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status snapshot.db

but etcdutl was not installed.

The system suggested:

sudo apt install etcd-server

I installed the package.

After installation, I still used the wrong filename once:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status snapshot.db
Error: stat snapshot.db: no such file or directory

Then I used the correct path:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status /opt/etcd-backup.db

This produced:

Error: open /opt/etcd-backup.db: permission denied

Again, the issue was file permissions.

So I used sudo:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

The result was:

┌──────────┬──────────┬────────────┬────────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ a633855e │     5973 │       1298 │     3.9 MB │
└──────────┴──────────┴────────────┴────────────┘

This confirmed that the snapshot was readable and contained etcd data.

13. Why create an nginx Deployment?

A backup is much more meaningful if we have something we can verify after restoring it.

At this point, I created an nginx Deployment.

My first attempt was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy nginx
error: required flag(s) "image" not set

The command was missing the container image.

I then tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy --image=nginx
error: exactly one NAME is required, got 0
See 'kubectl create deployment -h' for help and examples

Now the image was present, but the Deployment name was missing.

The correct command was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy nginx --image=nginx
deployment.apps/nginx created

This is a good beginner lesson:

kubectl create deployment <NAME> --image=<IMAGE>

So:

kubectl create deployment nginx --image=nginx

means:

create a Deployment;
call it nginx;
use the nginx container image.

14. Waiting for the nginx Pod

Immediately after creation:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME                        READY   STATUS              RESTARTS   AGE
pod/nginx-7f8fbb96d-9f7wm   0/1     ContainerCreating   0          4s

The Pod remained in:

ContainerCreating

I watched it:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -w

After waiting, I checked again.

The Pod eventually became:

nginx-7f8fbb96d-9f7wm   1/1   Running   0   3m16s

The describe output explained why it initially took time:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ---   ----               -------
  Normal  Scheduled  83s   default-scheduler  Successfully assigned default/nginx-7f8fbb96d-9f7wm to osboxes
  Normal  Pulling    82s   kubelet            spec.containers{nginx}: Pulling image "nginx"

The Pod had been scheduled successfully. The kubelet was simply pulling the image.

This is an important troubleshooting principle:

ContainerCreating does not automatically mean something is broken. Check the Pod events before changing anything.

Eventually:

k get po
NAME                    READY   STATUS    RESTARTS   AGE
nginx-7f8fbb96d-9f7wm   1/1     Running   0          3m16s

15. Take another backup after the nginx Deployment exists

This step is important because the first snapshot was taken before the nginx Deployment was created.

After nginx was running, I created another snapshot:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db

The snapshot was again:

3.9 MB

and the command ended with:

Snapshot saved at /opt/etcd-backup.db

I validated it:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Result:

┌──────────┬──────────┬────────────┬────────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ eb688bf4 │     6934 │        918 │     3.9 MB │
└──────────┴──────────┴────────────┴────────────┘

Notice that the revision changed.

The backup now represented a later state of the cluster.

16. Verify the state we want to recover

At this point:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME                        READY   STATUS    RESTARTS   AGE
pod/nginx-7f8fbb96d-9f7wm   1/1     Running   0          4m1s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   76m

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   1/1     1            1            4m2s

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-7f8fbb96d   1         1         1       4m2s

The important object is:

deployment.apps/nginx

It was healthy.

Now the test was simple:

If I delete nginx, can I use the etcd snapshot to recover the cluster state that existed when the snapshot was taken?

17. Delete nginx

I deleted the Deployment:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k delete deploy nginx
deployment.apps "nginx" deleted from default namespace

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   76m

The nginx Deployment was gone.

This gave us the desired recovery scenario:

BEFORE BACKUP
    |
    +-- nginx Deployment exists
    |
    v
TAKE ETCD SNAPSHOT
    |
    v
DELETE nginx
    |
    v
nginx no longer exists
    |
    v
RESTORE SNAPSHOT
    |
    v
expect previous cluster state

18. First restore mistake: wrong snapshot filename

I attempted:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --data-dir /opt/etcd/backup snapshot restore snapshot.db

The restore process immediately told me:

Error: open snapshot.db: no such file or directory

Again, the filename was wrong.

The actual snapshot was:

/opt/etcd-backup.db

This is a very common command-line mistake: creating a file with one name and later trying to restore a similarly named file.

The first restore attempt was therefore not an etcd failure. It was simply a path/filename error.

19. Restore the snapshot into a separate data directory

Instead of overwriting the active etcd data directory immediately, I restored the snapshot into a separate directory:

/var/lib/etcd-restore-from-backup

The command used was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot restore /opt/etcd-backup.db \
--data-dir=/var/lib/etcd-restore-from-backup

The output included:

Deprecated: Use `etcdutl snapshot restore` instead.

and then:

restoring snapshot

followed by:

Trimming membership information from the backend...

and:

added member

Finally:

restored snapshot

This was the actual successful restore of the snapshot into the new data directory.

20. Why use a separate restore directory?

The original etcd data directory was:

/var/lib/etcd

The restored copy was:

/var/lib/etcd-restore-from-backup

This distinction is important.

It allows the restored data to be prepared without immediately destroying the existing data directory.

For a beginner, think of it like restoring a database backup into a new folder first:

Original:
    /var/lib/etcd

Restored:
    /var/lib/etcd-restore-from-backup

Only after the restored data exists do we configure etcd to use it.

21. More path mistakes while inspecting the restored data

I attempted:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ cd /var/lib/data
bash: cd: /var/lib/data: No such file or directory

Again, the directory did not exist.

The restore directory was:

/var/lib/etcd-restore-from-backup

I also tried:

sudo cd /var/lib/etcd-restore-from-backup

and got:

sudo: cd: command not found
sudo: "cd" is a shell built-in command, it cannot be run directly.

This error is worth understanding.

cd is a shell built-in command. sudo normally executes an external command, so:

sudo cd ...

does not work.

Instead, because the directory required elevated permissions, I used:

sudo ls /var/lib/etcd-restore-from-backup

which showed:

member

That confirmed that the restore had created the expected etcd member directory structure.

22. Editing the static etcd manifest

The next step was to configure the static etcd pod to use the restored data directory.

The original manifest used:

--data-dir=/var/lib/etcd

The modified configuration used:

--data-dir=/var/lib/etcd-restore-from-backup

The corresponding hostPath was also changed to:

/var/lib/etcd-restore-from-backup

The resulting Pod description later confirmed:

--data-dir=/var/lib/etcd-restore-from-backup

and:

Path: /var/lib/etcd-restore-from-backup

This is how the restored database was connected to the etcd process.

23. Why moving the manifest files caused disruption

During the troubleshooting process, I temporarily moved the static pod manifests out of the manifest directory:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo mv * /tmp

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls -lrt
total 0

The directory was empty.

I then restored the manifests:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo mv /tmp/*.yaml .

and confirmed:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls -lrt
total 16
-rw------- 1 root root 3939 Aug 13 11:07 kube-apiserver.yaml
-rw------- 1 root root 3230 Aug 13 11:07 kube-controller-manager.yaml
-rw------- 1 root root 1726 Aug 13 11:07 kube-scheduler.yaml
-rw------- 1 root root 2596 Aug 13 12:37 etcd.yaml

This demonstrates how sensitive the static pod manifest directory is.

When the kubelet stops seeing a static pod manifest, the corresponding static pod can disappear.

For this reason, moving files in /etc/kubernetes/manifests should be done carefully.

24. Another small typo

I accidentally ran:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get oi
error: the server doesn't have a resource type "oi"

The intended command was clearly not oi.

The correct command for Pods was:

k get po

This is a harmless error, but it illustrates another important habit:

When kubectl reports that a resource type does not exist, first check the command spelling before assuming the Kubernetes API is broken.

25. The RBAC error after the manifest changes

After the static pod manipulation, this happened:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po
Error from server (Forbidden): pods is forbidden: User "kubernetes-admin" cannot list resource "pods" in API group "" in the namespace "default"

This was different from the earlier errors.

This time, the Kubernetes API server was responding, but the authenticated user was being denied access.

The command:

sudo k get po

was also not a solution:

sudo: k: command not found

The reason is that k was a shell alias, and aliases normally are not available to sudo in that way.

More importantly, sudo does not automatically solve Kubernetes RBAC problems.

I then checked the system namespace explicitly:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -n=kube-system

This returned the control-plane Pods.

That helped separate the problem from the actual etcd state.

26. etcd was now Pending

The output showed:

etcd-osboxes   0/1   Pending

I first tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po etcd-osboxes
Error from server (NotFound): pods "etcd-osboxes" not found

Again, the namespace mattered.

The correct command was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po -n=kube-system etcd-osboxes

The description showed the important configuration:

Status: Pending

but also:

Image: registry.k8s.io/etcd:3.6.8-0

and:

--data-dir=/var/lib/etcd-restore-from-backup

The Pod also showed the restored directory mounted:

/var/lib/etcd-restore-from-backup from etcd-data

and:

Path: /var/lib/etcd-restore-from-backup

The kubelet events showed:

Normal  Pulled   ...
Normal  Created  ...
Normal  Started  ...

So the container had actually been created and started even though the Kubernetes status remained Pending.

This is an important troubleshooting clue:

Do not rely on one status field alone. Read the full Pod description and events.

27. Trying to read etcd logs

I tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k logs etcd-osboxes -n kube-system
Error from server (BadRequest): container "etcd" in pod "etcd-osboxes" is not available

I also accidentally ran:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k logs

which returned:

error: expected 'logs [-f] [-p] (POD | TYPE/NAME) [-c CONTAINER]'.
POD or TYPE/NAME is a required argument for the logs command

The second error was simply because kubectl logs requires a Pod or workload name.

The first error was more interesting: Kubernetes did not consider the container available for normal log retrieval.

I then checked the Pod in wide format:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get pod etcd-osboxes -n kube-system -o wide
NAME           READY   STATUS    RESTARTS   AGE     IP       NODE      NOMINATED NODE   READINESS GATES
etcd-osboxes   0/1     Pending   0          4m58s   <none>   osboxes   <none>           <none>

The lack of a Pod IP was another indication that the Pod was not fully ready.

28. Inspecting kubelet

Since etcd was a static Pod, the kubelet was a key component to inspect.

I ran:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo systemctl status kubelet --no-pager

The result showed:

● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; disabled; preset: disabled)
     Active: active (running)

The kubelet was running.

The logs also showed repeated activity around the time of the problem.

At this point, rather than making more changes to the etcd manifest, I restarted kubelet:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo systemctl restart kubelet

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ systemctl daemon-reload

29. The recovery

After restarting kubelet, I checked the system namespace again:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -n=kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-74c68c8864-fjtnv   1/1     Running   5 (7m47s ago)   95m
calico-node-dzlcp                          1/1     Running   0               95m
coredns-589f44dc88-l522w                   1/1     Running   0               101m
coredns-589f44dc88-nzsh9                   1/1     Running   0               101m
etcd-osboxes                               1/1     Running   0               6m51s
kube-apiserver-osboxes                     1/1     Running   0               101m
kube-controller-manager-osboxes            1/1     Running   1 (7m32s ago)   101m
kube-proxy-d8krd                           1/1     Running   0               101m
kube-scheduler-osboxes                     1/1     Running   1               101m

This was the desired state.

Most importantly:

etcd-osboxes   1/1   Running

30. Confirming the restored etcd configuration

I then described the etcd Pod again:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po -n=kube-system etcd-osboxes

The important portion was:

Status: Running
IP: 10.0.2.15

The etcd container was:

State: Running
Ready: True
Restart Count: 0

And the command still showed:

--data-dir=/var/lib/etcd-restore-from-backup

The mounted volume was:

/var/lib/etcd-restore-from-backup from etcd-data

with the host path:

/var/lib/etcd-restore-from-backup

This confirmed that etcd was now running using the restored data directory.

31. What happened to the nginx Deployment?

The exercise was designed around the snapshot that was taken while nginx existed.

The important sequence was:

1. Create nginx Deployment
2. Wait for nginx to become Running
3. Take etcd snapshot
4. Delete nginx
5. Restore the snapshot
6. Point etcd at restored data
7. Restart kubelet
8. Confirm etcd is Running

The backup validation showed a snapshot revision of:

with:

918 total keys
3.9 MB total size

The restore operation successfully rebuilt an etcd data directory.

The final terminal state confirmed that etcd was successfully running from:

/var/lib/etcd-restore-from-backup

The terminal session provided here does not contain a final k get all after the successful etcd recovery showing nginx reappearing, so I would not claim that nginx restoration was independently verified from the terminal evidence.

That distinction matters.

The exercise successfully demonstrated:

snapshot creation;
snapshot validation;
snapshot restoration;
switching the static etcd Pod to restored data;
recovery of the etcd Pod.

The source terminal output does not provide a final explicit post-recovery k get all proving that the nginx Deployment was recreated from the restored snapshot.

32. The complete clean backup command

For this environment, the working backup command was:

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/etcd-backup.db

The important pieces are:

Endpoint

--endpoints=https://127.0.0.1:2379

This tells etcdctl where etcd is listening for client connections.

CA certificate

--cacert=/etc/kubernetes/pki/etcd/ca.crt

This allows the client to verify the etcd server certificate.

Client certificate

--cert=/etc/kubernetes/pki/etcd/server.crt

This identifies the client to the TLS-enabled etcd server.

Private key

--key=/etc/kubernetes/pki/etcd/server.key

This goes with the client certificate.

Snapshot command

snapshot save /opt/etcd-backup.db

This tells etcdctl to create a backend snapshot at the specified path.

33. The clean snapshot validation command

The session showed that etcdctl snapshot status is deprecated and recommended etcdutl.

The working validation command was:

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Example output from the session:

┌──────────┬──────────┬────────────┬────────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ eb688bf4 │     6934 │        918 │     3.9 MB │
└──────────┴──────────┴────────────┴────────────┘

This is useful because a backup file existing on disk does not, by itself, prove that it is a valid etcd snapshot.

34. The clean restore command

The restore performed successfully with:

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore-from-backup

The terminal explicitly reported:

Deprecated: Use `etcdutl snapshot restore` instead.

So for a newer workflow, prefer the modern etcdutl snapshot restore command rather than copying the deprecated command blindly.

35. Important troubleshooting lessons

Error 1: Wrong directory

cd /etc/kkubenetes/manifests

and:

cd /etc/kubenetes/manifests

failed because of spelling mistakes.

Fix

cd /etc/kubernetes/manifests

Error 2: Docker container not found

docker exec -it osboxes bash

returned:

No such container: osboxes

Lesson

A Kubernetes node name is not automatically a Docker container name.

Error 3: Permission denied reading etcd files

cat etcd.yaml

returned:

Permission denied

Fix

sudo cat etcd.yaml

The same principle applied to the protected etcd private key.

Error 4: Invalid etcdctl flag

---endpoint=...

returned:

Error: bad flag syntax

Fix

Use the correctly spelled flag:

--endpoints=...

Error 5: Private key permission denied

Error: open /etc/kubernetes/pki/etcd/server.key: permission denied

Fix

Run the etcd client command with sufficient privileges:

sudo etcdctl ...

Error 6: Wrong snapshot filename

snapshot.db

did not exist.

The actual file was:

/opt/etcd-backup.db

Lesson

Always verify the exact backup path before restoring.

Error 7: `etcdctl snapshot status` deprecated

The client reported:

Deprecated: Use `etcdutl snapshot status` instead.

Fix

Install/use etcdutl and run:

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Error 8: `etcdutl` not installed

The command:

etcdutl

was initially unavailable.

Fix

The system suggested installing etcd-server, which provided the utility in this environment.

Error 9: `sudo cd`

This failed:

sudo cd /var/lib/etcd-restore-from-backup

with:

sudo: cd: command not found

Lesson

cd is a shell built-in.

Use commands such as:

sudo ls /var/lib/etcd-restore-from-backup

or start a privileged shell if you genuinely need one.

Error 10: Pod name not found

This:

k describe po etcd-osboxes

returned:

pods "etcd-osboxes" not found

because the command defaulted to the default namespace.

Fix

k describe po -n=kube-system etcd-osboxes

Error 11: `kubectl logs` without a Pod

k logs

returned:

POD or TYPE/NAME is a required argument for the logs command

Fix

Provide the resource:

k logs <pod-name>

and specify the namespace when needed.

Error 12: `kubectl` Forbidden

After the control-plane changes:

Error from server (Forbidden): pods is forbidden:
User "kubernetes-admin" cannot list resource "pods"

This was an authorization/RBAC problem rather than a shell permission problem.

sudo did not solve it because Kubernetes authorization is separate from Linux sudo.

Error 13: etcd remained Pending

The etcd Pod stayed:

0/1 Pending

even though the container had been created and started according to the events.

The investigation included:

k describe po -n=kube-system etcd-osboxes

and:

sudo systemctl status kubelet --no-pager

The practical recovery step that resolved the state in this lab was:

sudo systemctl restart kubelet

After that:

etcd-osboxes   1/1   Running

36. Why static Pods matter in this exercise

The etcd Pod was not created with a normal Deployment such as:

kind: Deployment

Instead, it was defined by:

/etc/kubernetes/manifests/etcd.yaml

That makes it a static Pod.

The kubelet watches the manifest directory and manages the Pod locally.

This explains several things that happened during the exercise:

changing etcd.yaml affected etcd;
temporarily moving manifests out of the directory affected the control plane;
restarting kubelet caused it to re-read and reconcile the manifests;
etcd could be controlled without creating a normal Kubernetes Deployment.

For kubeadm clusters, understanding static Pods is essential when troubleshooting the control plane.

37. A safer mental model for etcd restore

Do not think of restore as:

"copy backup file over etcd"

A better mental model is:

ETCD SNAPSHOT
     |
     v
restore snapshot
     |
     v
new etcd data directory
     |
     v
configure etcd to use restored directory
     |
     v
kubelet recreates/restarts static etcd Pod
     |
     v
etcd becomes healthy
     |
     v
Kubernetes API becomes available against restored state

This is why the exercise used:

/var/lib/etcd-restore-from-backup

instead of immediately destroying:

/var/lib/etcd

38. What I would do differently next time

The lab worked, but it also exposed several areas where the workflow could be cleaner.

1. Check the exact paths first

Before typing a long command:

ls -l /etc/kubernetes/pki/etcd/
ls -l /opt/etcd-backup.db

This avoids guessing filenames and permissions.

2. Inspect the manifest before changing it

First:

sudo cp /etc/kubernetes/manifests/etcd.yaml /root/etcd.yaml.backup

Then edit the manifest.

That gives you a known-good copy if something goes wrong.

3. Prefer the current etcd utilities

The terminal explicitly showed that:

etcdctl snapshot status

is deprecated and recommended:

etcdutl snapshot status

Likewise, the restore output recommended:

etcdutl snapshot restore

4. Avoid unnecessary movement of all manifests

This command:

sudo mv * /tmp

is powerful and potentially disruptive.

Moving all control-plane manifests affects more than etcd.

A safer approach is to modify only the required manifest and keep backups of the original.

5. Verify each stage

After every major step:

k get pods -A

or:

k get pods -n kube-system

and inspect:

k describe pod <pod> -n kube-system

if anything looks abnormal.

39. A compact recovery checklist

For a kubeadm single-control-plane lab similar to this one:

Identify etcd

k get pods -n kube-system | grep etcd

Inspect the manifest

cd /etc/kubernetes/manifests
sudo cat etcd.yaml

Identify:

--data-dir
--listen-client-urls
--cert-file
--key-file
--trusted-ca-file

Create snapshot

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/etcd-backup.db

Validate snapshot

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Restore to a separate directory

sudo etcdutl snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore-from-backup

Configure etcd to use the restored directory

Update:

--data-dir=/var/lib/etcd-restore-from-backup

and the corresponding hostPath.

Restart/reconcile kubelet if required

sudo systemctl restart kubelet

Verify

k get pods -n kube-system

Look for:

etcd-osboxes   1/1   Running

40. What this lab actually taught me

The most valuable part of this exercise was not memorizing one command.

It was learning how the pieces connect:

Kubernetes
    |
    +-- kube-apiserver
    |
    +-- kube-controller-manager
    |
    +-- kube-scheduler
    |
    +-- kubelet
            |
            +-- static Pod manifests
                    |
                    +-- etcd
                            |
                            +-- /var/lib/etcd

The backup process interacted directly with etcd:

etcd
  |
  +-- TLS certificates
  |
  +-- etcdctl
  |
  +-- snapshot
  |
  +-- /opt/etcd-backup.db

The restore process then created:

/var/lib/etcd-restore-from-backup

and the static Pod was changed to consume that restored data.

Finally, kubelet reconciled the static Pod and etcd returned to:

1/1 Running

41. Final takeaway

An etcd backup is not just another Kubernetes YAML export.

The core workflow is:

1. Verify etcd is healthy.
2. Find the etcd endpoint and TLS certificates.
3. Take an etcd snapshot.
4. Validate the snapshot.
5. Restore the snapshot into a separate data directory.
6. Configure the etcd static Pod to use the restored data.
7. Let kubelet reconcile the static Pod.
8. Verify etcd and the Kubernetes control plane.

The terminal session also demonstrated something that documentation often hides: the commands do not always work on the first attempt.

There were:

spelling mistakes;
incorrect flags;
permission problems;
wrong filenames;
deprecated commands;
missing utilities;
namespace mistakes;
RBAC errors;
attempts to use sudo where it did not apply;
a static etcd Pod that remained Pending;
and finally a kubelet restart that brought etcd back to Running.

Those failures were not separate from the learning experience. They were the learning experience.

The most useful troubleshooting habit from this lab is simple:

When Kubernetes gives you an error, do not immediately change random things. Read the error, identify which layer produced it, inspect the relevant component, make the smallest reasonable correction, and verify the result.

That is the difference between memorizing Kubernetes commands and actually learning how the control plane works.

Terminal command reference from the lab

Cluster health

k get pods -A
k get pods -A -w
k get nodes
k get all
k get all -A

Static Pod manifests

cd /etc/kubernetes/manifests
ls
sudo cat etcd.yaml

Install tooling

sudo apt install etcd-client
sudo apt install etcd-server

Configure etcdctl

export ETCDCTL_API=3

Backup

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/etcd-backup.db

Validate

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Test workload

k create deploy nginx --image=nginx
k get po
k get po -w
k describe po nginx-7f8fbb96d-9f7wm
k get all

Delete workload

k delete deploy nginx

Restore

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore-from-backup

Troubleshoot etcd

k get po -n=kube-system
k describe po -n=kube-system etcd-osboxes
k get pod etcd-osboxes -n kube-system -o wide
k logs etcd-osboxes -n kube-system
sudo systemctl status kubelet --no-pager
sudo systemctl restart kubelet

Final verification

k get po -n=kube-system
k describe po -n=kube-system etcd-osboxes

A beginner-friendly, hands-on walkthrough of backing up and restoring etcd in a kubeadm-based Kubernetes cluster — including the mistakes, errors, debugging steps, and the terminal commands that actually happened during the lab.

Introduction

If you are learning Kubernetes, you will eventually hear a simple but important statement:

etcd is the database of a Kubernetes cluster.

That description is short, but it has a major consequence: if the etcd data is lost or corrupted, the Kubernetes control plane can lose the information it needs to know what exists in the cluster.

In this lab, I practiced an end-to-end etcd backup and restore workflow on a single-node kubeadm cluster running on a machine named osboxes.

The exercise covered:

identifying the etcd pod and confirming that the control plane was healthy;
locating the static etcd manifest;
inspecting the etcd configuration;
installing etcdctl and etcdutl;
creating an etcd snapshot;
validating the snapshot;
deliberately creating an nginx Deployment so there was something meaningful to restore;
deleting the Deployment;
restoring the etcd snapshot into a separate data directory;
changing the static etcd manifest to use the restored data;
troubleshooting an etcd pod that remained Pending;
restarting kubelet;
confirming that etcd eventually returned to Running.

The important part of this exercise was not simply getting a successful backup command. It was understanding what each error meant and how to reason from the evidence in the terminal.

1. First: What is etcd?

Before touching commands, it helps to understand what we are backing up.

Kubernetes has many components, but the important relationship for this exercise is roughly:

kubectl
   |
   v
kube-apiserver
   |
   v
  etcd
   |
   +---- Kubernetes cluster state

The API server is the main interface through which Kubernetes clients interact with the cluster. etcd is the persistent key-value store used by Kubernetes to store cluster state.

That state includes information such as:

Kubernetes objects;
Deployments;
Pods and their specifications;
Services;
ConfigMaps;
Secrets;
cluster configuration and metadata.

A useful mental model for a beginner is:

The Kubernetes API is how you ask Kubernetes about the cluster; etcd is where much of that cluster state is persisted.

This is why backing up etcd is fundamentally different from simply exporting Kubernetes YAML.

2. Verify that the cluster is healthy first

Before taking a backup, I first checked the cluster.

Terminal

┌──(osboxes㉿osboxes)-[~]
└─$ k get pods -A
NAMESPACE     NAME                                       READY   STATUS              RESTARTS   AGE
kube-system   calico-kube-controllers-74c68c8864-fjtnv   0/1     ContainerCreating   0          3m10s
kube-system   calico-node-dzlcp                          0/1     Init:2/3             0          3m10s
kube-system   coredns-589f44dc88-l522w                   0/1     ContainerCreating   0          8m56s
kube-system   coredns-589f44dc88-nzsh9                   0/1     ContainerCreating   0          8m56s
kube-system   etcd-osboxes                               1/1     Running             0          9m6s
kube-system   kube-apiserver-osboxes                     1/1     Running             0          9m9s
kube-system   kube-controller-manager-osboxes            1/1     Running             0          9m5s
kube-system   kube-proxy-d8krd                           1/1     Running             0          8m56s
kube-system   kube-scheduler-osboxes                     1/1     Running             0          9m8s

The most important observation here is:

etcd-osboxes  1/1  Running

The API server, controller manager, scheduler and kube-proxy were also running.

Some networking components were still starting, so I watched the pods:

┌──(osboxes㉿osboxes)-[~]
└─$ k get pods -A -w

Eventually CoreDNS and Calico became ready.

I then checked the node:

┌──(osboxes㉿osboxes)-[~]
└─$ k get nodes
NAME      STATUS   ROLES           AGE   VERSION
osboxes   Ready    control-plane   10m   v1.36.3

At this point the single control-plane node was Ready.

The initial cluster state is important because a backup should be taken from a functioning etcd instance rather than blindly assuming that etcd is healthy.

3. An important distinction: Kubernetes YAML backup vs etcd backup

Before working directly with etcd, I also ran:

┌──(osboxes㉿osboxes)-[~]
└─$ k get all -A -o yaml > backup.yaml

This produced:

┌──(osboxes㉿osboxes)-[~]
└─$ ls
backup.yaml  Documents  kubectl  Pictures  Public  Videos
Desktop      Downloads  Music    Projects  Templates  VM-Share

This is useful, but it is not the same thing as an etcd snapshot.

The command:

k get all -A -o yaml > backup.yaml

exports Kubernetes resources returned by kubectl get all.

An etcd snapshot, on the other hand, is a snapshot of the etcd backend itself.

For an etcd disaster-recovery exercise, the etcd snapshot is the important artifact.

4. Finding the etcd static pod

This cluster was created with kubeadm, so etcd was running as a static pod.

I initially made a typo:

┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kkubenetes/manifests
bash: cd: /etc/kkubenetes/manifests: No such file or directory

The problem was simply the spelling:

kkubenetes

instead of:

kubernetes

I then made another typo:

┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kubenetes/manifests
bash: cd: /etc/kubenetes/manifests: No such file or directory

Again, the correct directory is:

/etc/kubernetes/manifests

Eventually:

┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kubernetes/manifests

and:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls
etcd.yaml
kube-apiserver.yaml
kube-controller-manager.yaml
kube-scheduler.yaml

This directory is extremely important on a kubeadm control-plane node.

The files in /etc/kubernetes/manifests are static pod manifests. The kubelet watches this directory and makes sure the described control-plane containers are running.

That means editing etcd.yaml is effectively changing how the control-plane etcd process is launched.

5. Another wrong assumption: trying to use Docker

I also tried:

┌──(osboxes㉿osboxes)-[~]
└─$ docker exec -it osboxes bash
Error response from daemon: No such container: osboxes

This failed because the Kubernetes node name osboxes is not necessarily a Docker container name.

In this setup, the node was running Kubernetes with containerd, and the etcd pod later showed a container ID beginning with:

containerd://

So the important lesson is:

Do not assume that a Kubernetes node or Pod name is a Docker container name.

If you need to inspect a Kubernetes container, first determine which container runtime the node is using.

6. Inspecting the etcd manifest

The first attempt to read the manifest without elevated privileges also failed:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ cat etcd.yaml
cat: etcd.yaml: Permission denied

The solution was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo cat etcd.yaml

The manifest revealed several important values.

For example:

- --advertise-client-urls=https://10.0.2.15:2379
- --cert-file=/etc/kubernetes/pki/etcd/server.crt
- --client-cert-auth=true
- --data-dir=/var/lib/etcd
- --key-file=/etc/kubernetes/pki/etcd/server.key
- --listen-client-urls=https://127.0.0.1:2379,https://10.0.2.15:2379

It also showed that the etcd data directory was originally:

/var/lib/etcd

and the TLS certificates were under:

/etc/kubernetes/pki/etcd/

This information is essential because etcdctl needs to connect to the correct endpoint and authenticate with the appropriate certificates.

7. Install etcdctl

The machine did not initially have the etcd client installed, so I installed it:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo apt install etcd-client

The package installed successfully:

Installing:
  etcd-client

Summary:
  Upgrading: 0, Installing: 1, Removing: 0, Not Upgrading: 205

I then checked the snapshot command:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ETCDCTL_API=3 etcdctl snapshot

The output showed:

COMMANDS:
    restore  Restores an etcd member snapshot to an etcd directory
    save     Stores an etcd node backend snapshot to a given file
    status   [deprecated] Gets backend snapshot status of a given file

The key command for the backup is:

etcdctl snapshot save

8. ETCDCTL_API=3

I also explicitly exported:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ export ETCDCTL_API=3

For the lab, this made it explicit that etcdctl should use the v3 API.

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl snapshot

again showed the snapshot subcommands.

For a beginner, the important idea is:

ETCDCTL_API=3

selects the etcd API version used by the client.

9. Creating the first etcd snapshot — and making a typo

I attempted the backup with:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl ---endpoint=https://127.0.0.1:2379,https://10.0.2.15:2379 \
> --cacert=/etc/kubernetes/pki/etcd/ca.crt \
> --cert=/etc/kubernetes/pki/etcd/server.crt \
> --key=/etc/kubernetes/pki/etcd/server.key \
> snapshot save /opt/etcd-backup.db

The error was:

Error: bad flag syntax: ---endpoint=https://127.0.0.1:2379,https://10.0.2.15:2379

The mistake was the number of hyphens.

I typed:

---endpoint

instead of:

--endpoints

There were actually two problems worth noticing:

There were three hyphens.
The correct flag is --endpoints, plural.

This is a classic CLI problem: the command can look almost correct to a human while being completely invalid to the parser.

10. The next error: permission denied

I corrected the flag and tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db

The command reached the certificate files but failed with:

Error: open /etc/kubernetes/pki/etcd/server.key: permission denied

This made sense.

The etcd private key is protected, and the regular user did not have permission to read it.

The solution was to run the command with sudo:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db

This time the backup succeeded.

11. The successful etcd backup

The successful command produced:

{"level":"info","ts":"2026-08-13T12:11:59.649233-0400","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/opt/etcd-backup.db.part"}
{"level":"info","ts":"2026-08-13T12:11:59.660384-0400","logger":"client","caller":"v3/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2026-08-13T12:11:59.660467-0400","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2026-08-13T12:11:59.831074-0400","logger":"client","caller":"v3/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2026-08-13T12:11:59.894020-0400","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"3.9 MB","took":"now"}
{"level":"info","ts":"2026-08-13T12:11:59.894158-0400","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/opt/etcd-backup.db"}
Snapshot saved at /opt/etcd-backup.db

The resulting backup file was:

/opt/etcd-backup.db

The snapshot was approximately:

3.9 MB

This was the first major milestone.

12. Validating the snapshot

I initially tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl --write-out=table snapshot status snapshot.db

The client responded:

Deprecated: Use `etcdutl snapshot status` instead.

Error: stat snapshot.db: no such file or directory

There were two lessons here.

Lesson 1: The filename was wrong

The backup was saved as:

/opt/etcd-backup.db

not:

snapshot.db

Lesson 2: The command was deprecated

The output explicitly recommended:

etcdutl snapshot status

So I attempted:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status snapshot.db

but etcdutl was not installed.

The system suggested:

sudo apt install etcd-server

I installed the package.

After installation, I still used the wrong filename once:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status snapshot.db
Error: stat snapshot.db: no such file or directory

Then I used the correct path:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status /opt/etcd-backup.db

This produced:

Error: open /opt/etcd-backup.db: permission denied

Again, the issue was file permissions.

So I used sudo:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

The result was:

┌──────────┬──────────┬────────────┬────────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ a633855e │     5973 │       1298 │     3.9 MB │
└──────────┴──────────┴────────────┴────────────┘

This confirmed that the snapshot was readable and contained etcd data.

13. Why create an nginx Deployment?

A backup is much more meaningful if we have something we can verify after restoring it.

At this point, I created an nginx Deployment.

My first attempt was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy nginx
error: required flag(s) "image" not set

The command was missing the container image.

I then tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy --image=nginx
error: exactly one NAME is required, got 0
See 'kubectl create deployment -h' for help and examples

Now the image was present, but the Deployment name was missing.

The correct command was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy nginx --image=nginx
deployment.apps/nginx created

This is a good beginner lesson:

kubectl create deployment <NAME> --image=<IMAGE>

So:

kubectl create deployment nginx --image=nginx

means:

create a Deployment;
call it nginx;
use the nginx container image.

14. Waiting for the nginx Pod

Immediately after creation:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME                        READY   STATUS              RESTARTS   AGE
pod/nginx-7f8fbb96d-9f7wm   0/1     ContainerCreating   0          4s

The Pod remained in:

ContainerCreating

I watched it:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -w

After waiting, I checked again.

The Pod eventually became:

nginx-7f8fbb96d-9f7wm   1/1   Running   0   3m16s

The describe output explained why it initially took time:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ---   ----               -------
  Normal  Scheduled  83s   default-scheduler  Successfully assigned default/nginx-7f8fbb96d-9f7wm to osboxes
  Normal  Pulling    82s   kubelet            spec.containers{nginx}: Pulling image "nginx"

The Pod had been scheduled successfully. The kubelet was simply pulling the image.

This is an important troubleshooting principle:

ContainerCreating does not automatically mean something is broken. Check the Pod events before changing anything.

Eventually:

k get po
NAME                    READY   STATUS    RESTARTS   AGE
nginx-7f8fbb96d-9f7wm   1/1     Running   0          3m16s

15. Take another backup after the nginx Deployment exists

This step is important because the first snapshot was taken before the nginx Deployment was created.

After nginx was running, I created another snapshot:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db

The snapshot was again:

3.9 MB

and the command ended with:

Snapshot saved at /opt/etcd-backup.db

I validated it:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Result:

┌──────────┬──────────┬────────────┬────────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ eb688bf4 │     6934 │        918 │     3.9 MB │
└──────────┴──────────┴────────────┴────────────┘

Notice that the revision changed.

The backup now represented a later state of the cluster.

16. Verify the state we want to recover

At this point:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME                        READY   STATUS    RESTARTS   AGE
pod/nginx-7f8fbb96d-9f7wm   1/1     Running   0          4m1s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   76m

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   1/1     1            1            4m2s

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-7f8fbb96d   1         1         1       4m2s

The important object is:

deployment.apps/nginx

It was healthy.

Now the test was simple:

If I delete nginx, can I use the etcd snapshot to recover the cluster state that existed when the snapshot was taken?

17. Delete nginx

I deleted the Deployment:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k delete deploy nginx
deployment.apps "nginx" deleted from default namespace

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   76m

The nginx Deployment was gone.

This gave us the desired recovery scenario:

BEFORE BACKUP
    |
    +-- nginx Deployment exists
    |
    v
TAKE ETCD SNAPSHOT
    |
    v
DELETE nginx
    |
    v
nginx no longer exists
    |
    v
RESTORE SNAPSHOT
    |
    v
expect previous cluster state

18. First restore mistake: wrong snapshot filename

I attempted:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --data-dir /opt/etcd/backup snapshot restore snapshot.db

The restore process immediately told me:

Error: open snapshot.db: no such file or directory

Again, the filename was wrong.

The actual snapshot was:

/opt/etcd-backup.db

This is a very common command-line mistake: creating a file with one name and later trying to restore a similarly named file.

The first restore attempt was therefore not an etcd failure. It was simply a path/filename error.

19. Restore the snapshot into a separate data directory

Instead of overwriting the active etcd data directory immediately, I restored the snapshot into a separate directory:

/var/lib/etcd-restore-from-backup

The command used was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot restore /opt/etcd-backup.db \
--data-dir=/var/lib/etcd-restore-from-backup

The output included:

Deprecated: Use `etcdutl snapshot restore` instead.

and then:

restoring snapshot

followed by:

Trimming membership information from the backend...

and:

added member

Finally:

restored snapshot

This was the actual successful restore of the snapshot into the new data directory.

20. Why use a separate restore directory?

The original etcd data directory was:

/var/lib/etcd

The restored copy was:

/var/lib/etcd-restore-from-backup

This distinction is important.

It allows the restored data to be prepared without immediately destroying the existing data directory.

For a beginner, think of it like restoring a database backup into a new folder first:

Original:
    /var/lib/etcd

Restored:
    /var/lib/etcd-restore-from-backup

Only after the restored data exists do we configure etcd to use it.

21. More path mistakes while inspecting the restored data

I attempted:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ cd /var/lib/data
bash: cd: /var/lib/data: No such file or directory

Again, the directory did not exist.

The restore directory was:

/var/lib/etcd-restore-from-backup

I also tried:

sudo cd /var/lib/etcd-restore-from-backup

and got:

sudo: cd: command not found
sudo: "cd" is a shell built-in command, it cannot be run directly.

This error is worth understanding.

cd is a shell built-in command. sudo normally executes an external command, so:

sudo cd ...

does not work.

Instead, because the directory required elevated permissions, I used:

sudo ls /var/lib/etcd-restore-from-backup

which showed:

member

That confirmed that the restore had created the expected etcd member directory structure.

22. Editing the static etcd manifest

The next step was to configure the static etcd pod to use the restored data directory.

The original manifest used:

--data-dir=/var/lib/etcd

The modified configuration used:

--data-dir=/var/lib/etcd-restore-from-backup

The corresponding hostPath was also changed to:

/var/lib/etcd-restore-from-backup

The resulting Pod description later confirmed:

--data-dir=/var/lib/etcd-restore-from-backup

and:

Path: /var/lib/etcd-restore-from-backup

This is how the restored database was connected to the etcd process.

23. Why moving the manifest files caused disruption

During the troubleshooting process, I temporarily moved the static pod manifests out of the manifest directory:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo mv * /tmp

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls -lrt
total 0

The directory was empty.

I then restored the manifests:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo mv /tmp/*.yaml .

and confirmed:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls -lrt
total 16
-rw------- 1 root root 3939 Aug 13 11:07 kube-apiserver.yaml
-rw------- 1 root root 3230 Aug 13 11:07 kube-controller-manager.yaml
-rw------- 1 root root 1726 Aug 13 11:07 kube-scheduler.yaml
-rw------- 1 root root 2596 Aug 13 12:37 etcd.yaml

This demonstrates how sensitive the static pod manifest directory is.

When the kubelet stops seeing a static pod manifest, the corresponding static pod can disappear.

For this reason, moving files in /etc/kubernetes/manifests should be done carefully.

24. Another small typo

I accidentally ran:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get oi
error: the server doesn't have a resource type "oi"

The intended command was clearly not oi.

The correct command for Pods was:

k get po

This is a harmless error, but it illustrates another important habit:

When kubectl reports that a resource type does not exist, first check the command spelling before assuming the Kubernetes API is broken.

25. The RBAC error after the manifest changes

After the static pod manipulation, this happened:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po
Error from server (Forbidden): pods is forbidden: User "kubernetes-admin" cannot list resource "pods" in API group "" in the namespace "default"

This was different from the earlier errors.

This time, the Kubernetes API server was responding, but the authenticated user was being denied access.

The command:

sudo k get po

was also not a solution:

sudo: k: command not found

The reason is that k was a shell alias, and aliases normally are not available to sudo in that way.

More importantly, sudo does not automatically solve Kubernetes RBAC problems.

I then checked the system namespace explicitly:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -n=kube-system

This returned the control-plane Pods.

That helped separate the problem from the actual etcd state.

26. etcd was now Pending

The output showed:

etcd-osboxes   0/1   Pending

I first tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po etcd-osboxes
Error from server (NotFound): pods "etcd-osboxes" not found

Again, the namespace mattered.

The correct command was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po -n=kube-system etcd-osboxes

The description showed the important configuration:

Status: Pending

but also:

Image: registry.k8s.io/etcd:3.6.8-0

and:

--data-dir=/var/lib/etcd-restore-from-backup

The Pod also showed the restored directory mounted:

/var/lib/etcd-restore-from-backup from etcd-data

and:

Path: /var/lib/etcd-restore-from-backup

The kubelet events showed:

Normal  Pulled   ...
Normal  Created  ...
Normal  Started  ...

So the container had actually been created and started even though the Kubernetes status remained Pending.

This is an important troubleshooting clue:

Do not rely on one status field alone. Read the full Pod description and events.

27. Trying to read etcd logs

I tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k logs etcd-osboxes -n kube-system
Error from server (BadRequest): container "etcd" in pod "etcd-osboxes" is not available

I also accidentally ran:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k logs

which returned:

error: expected 'logs [-f] [-p] (POD | TYPE/NAME) [-c CONTAINER]'.
POD or TYPE/NAME is a required argument for the logs command

The second error was simply because kubectl logs requires a Pod or workload name.

The first error was more interesting: Kubernetes did not consider the container available for normal log retrieval.

I then checked the Pod in wide format:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get pod etcd-osboxes -n kube-system -o wide
NAME           READY   STATUS    RESTARTS   AGE     IP       NODE      NOMINATED NODE   READINESS GATES
etcd-osboxes   0/1     Pending   0          4m58s   <none>   osboxes   <none>           <none>

The lack of a Pod IP was another indication that the Pod was not fully ready.

28. Inspecting kubelet

Since etcd was a static Pod, the kubelet was a key component to inspect.

I ran:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo systemctl status kubelet --no-pager

The result showed:

● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; disabled; preset: disabled)
     Active: active (running)

The kubelet was running.

The logs also showed repeated activity around the time of the problem.

At this point, rather than making more changes to the etcd manifest, I restarted kubelet:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo systemctl restart kubelet

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ systemctl daemon-reload

29. The recovery

After restarting kubelet, I checked the system namespace again:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -n=kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-74c68c8864-fjtnv   1/1     Running   5 (7m47s ago)   95m
calico-node-dzlcp                          1/1     Running   0               95m
coredns-589f44dc88-l522w                   1/1     Running   0               101m
coredns-589f44dc88-nzsh9                   1/1     Running   0               101m
etcd-osboxes                               1/1     Running   0               6m51s
kube-apiserver-osboxes                     1/1     Running   0               101m
kube-controller-manager-osboxes            1/1     Running   1 (7m32s ago)   101m
kube-proxy-d8krd                           1/1     Running   0               101m
kube-scheduler-osboxes                     1/1     Running   1               101m

This was the desired state.

Most importantly:

etcd-osboxes   1/1   Running

30. Confirming the restored etcd configuration

I then described the etcd Pod again:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po -n=kube-system etcd-osboxes

The important portion was:

Status: Running
IP: 10.0.2.15

The etcd container was:

State: Running
Ready: True
Restart Count: 0

And the command still showed:

--data-dir=/var/lib/etcd-restore-from-backup

The mounted volume was:

/var/lib/etcd-restore-from-backup from etcd-data

with the host path:

/var/lib/etcd-restore-from-backup

This confirmed that etcd was now running using the restored data directory.

31. What happened to the nginx Deployment?

The exercise was designed around the snapshot that was taken while nginx existed.

The important sequence was:

1. Create nginx Deployment
2. Wait for nginx to become Running
3. Take etcd snapshot
4. Delete nginx
5. Restore the snapshot
6. Point etcd at restored data
7. Restart kubelet
8. Confirm etcd is Running

The backup validation showed a snapshot revision of:

with:

918 total keys
3.9 MB total size

The restore operation successfully rebuilt an etcd data directory.

The final terminal state confirmed that etcd was successfully running from:

/var/lib/etcd-restore-from-backup

That distinction matters.

The exercise successfully demonstrated:

snapshot creation;
snapshot validation;
snapshot restoration;
switching the static etcd Pod to restored data;
recovery of the etcd Pod.

The source terminal output does not provide a final explicit post-recovery k get all proving that the nginx Deployment was recreated from the restored snapshot.

32. The complete clean backup command

For this environment, the working backup command was:

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/etcd-backup.db

The important pieces are:

Endpoint

--endpoints=https://127.0.0.1:2379

This tells etcdctl where etcd is listening for client connections.

CA certificate

--cacert=/etc/kubernetes/pki/etcd/ca.crt

This allows the client to verify the etcd server certificate.

Client certificate

--cert=/etc/kubernetes/pki/etcd/server.crt

This identifies the client to the TLS-enabled etcd server.

Private key

--key=/etc/kubernetes/pki/etcd/server.key

This goes with the client certificate.

Snapshot command

snapshot save /opt/etcd-backup.db

This tells etcdctl to create a backend snapshot at the specified path.

33. The clean snapshot validation command

The session showed that etcdctl snapshot status is deprecated and recommended etcdutl.

The working validation command was:

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Example output from the session:

┌──────────┬──────────┬────────────┬────────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ eb688bf4 │     6934 │        918 │     3.9 MB │
└──────────┴──────────┴────────────┴────────────┘

This is useful because a backup file existing on disk does not, by itself, prove that it is a valid etcd snapshot.

34. The clean restore command

The restore performed successfully with:

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore-from-backup

The terminal explicitly reported:

Deprecated: Use `etcdutl snapshot restore` instead.

So for a newer workflow, prefer the modern etcdutl snapshot restore command rather than copying the deprecated command blindly.

35. Important troubleshooting lessons

Error 1: Wrong directory

cd /etc/kkubenetes/manifests

and:

cd /etc/kubenetes/manifests

failed because of spelling mistakes.

Fix

cd /etc/kubernetes/manifests

Error 2: Docker container not found

docker exec -it osboxes bash

returned:

No such container: osboxes

Lesson

A Kubernetes node name is not automatically a Docker container name.

Error 3: Permission denied reading etcd files

cat etcd.yaml

returned:

Permission denied

Fix

sudo cat etcd.yaml

The same principle applied to the protected etcd private key.

Error 4: Invalid etcdctl flag

---endpoint=...

returned:

Error: bad flag syntax

Fix

Use the correctly spelled flag:

--endpoints=...

Error 5: Private key permission denied

Error: open /etc/kubernetes/pki/etcd/server.key: permission denied

Fix

Run the etcd client command with sufficient privileges:

sudo etcdctl ...

Error 6: Wrong snapshot filename

snapshot.db

did not exist.

The actual file was:

/opt/etcd-backup.db

Lesson

Always verify the exact backup path before restoring.

Error 7: `etcdctl snapshot status` deprecated

The client reported:

Deprecated: Use `etcdutl snapshot status` instead.

Fix

Install/use etcdutl and run:

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Error 8: `etcdutl` not installed

The command:

etcdutl

was initially unavailable.

Fix

The system suggested installing etcd-server, which provided the utility in this environment.

Error 9: `sudo cd`

This failed:

sudo cd /var/lib/etcd-restore-from-backup

with:

sudo: cd: command not found

Lesson

cd is a shell built-in.

Use commands such as:

sudo ls /var/lib/etcd-restore-from-backup

or start a privileged shell if you genuinely need one.

Error 10: Pod name not found

This:

k describe po etcd-osboxes

returned:

pods "etcd-osboxes" not found

because the command defaulted to the default namespace.

Fix

k describe po -n=kube-system etcd-osboxes

Error 11: `kubectl logs` without a Pod

k logs

returned:

POD or TYPE/NAME is a required argument for the logs command

Fix

Provide the resource:

k logs <pod-name>

and specify the namespace when needed.

Error 12: `kubectl` Forbidden

After the control-plane changes:

Error from server (Forbidden): pods is forbidden:
User "kubernetes-admin" cannot list resource "pods"

This was an authorization/RBAC problem rather than a shell permission problem.

sudo did not solve it because Kubernetes authorization is separate from Linux sudo.

Error 13: etcd remained Pending

The etcd Pod stayed:

0/1 Pending

even though the container had been created and started according to the events.

The investigation included:

k describe po -n=kube-system etcd-osboxes

and:

sudo systemctl status kubelet --no-pager

The practical recovery step that resolved the state in this lab was:

sudo systemctl restart kubelet

After that:

etcd-osboxes   1/1   Running

36. Why static Pods matter in this exercise

The etcd Pod was not created with a normal Deployment such as:

kind: Deployment

Instead, it was defined by:

/etc/kubernetes/manifests/etcd.yaml

That makes it a static Pod.

The kubelet watches the manifest directory and manages the Pod locally.

This explains several things that happened during the exercise:

changing etcd.yaml affected etcd;
temporarily moving manifests out of the directory affected the control plane;
restarting kubelet caused it to re-read and reconcile the manifests;
etcd could be controlled without creating a normal Kubernetes Deployment.

For kubeadm clusters, understanding static Pods is essential when troubleshooting the control plane.

37. A safer mental model for etcd restore

Do not think of restore as:

"copy backup file over etcd"

A better mental model is:

ETCD SNAPSHOT
     |
     v
restore snapshot
     |
     v
new etcd data directory
     |
     v
configure etcd to use restored directory
     |
     v
kubelet recreates/restarts static etcd Pod
     |
     v
etcd becomes healthy
     |
     v
Kubernetes API becomes available against restored state

This is why the exercise used:

/var/lib/etcd-restore-from-backup

instead of immediately destroying:

/var/lib/etcd

38. What I would do differently next time

The lab worked, but it also exposed several areas where the workflow could be cleaner.

1. Check the exact paths first

Before typing a long command:

ls -l /etc/kubernetes/pki/etcd/
ls -l /opt/etcd-backup.db

This avoids guessing filenames and permissions.

2. Inspect the manifest before changing it

First:

sudo cp /etc/kubernetes/manifests/etcd.yaml /root/etcd.yaml.backup

Then edit the manifest.

That gives you a known-good copy if something goes wrong.

3. Prefer the current etcd utilities

The terminal explicitly showed that:

etcdctl snapshot status

is deprecated and recommended:

etcdutl snapshot status

Likewise, the restore output recommended:

etcdutl snapshot restore

4. Avoid unnecessary movement of all manifests

This command:

sudo mv * /tmp

is powerful and potentially disruptive.

Moving all control-plane manifests affects more than etcd.

A safer approach is to modify only the required manifest and keep backups of the original.

5. Verify each stage

After every major step:

k get pods -A

or:

k get pods -n kube-system

and inspect:

k describe pod <pod> -n kube-system

if anything looks abnormal.

39. A compact recovery checklist

For a kubeadm single-control-plane lab similar to this one:

Identify etcd

k get pods -n kube-system | grep etcd

Inspect the manifest

cd /etc/kubernetes/manifests
sudo cat etcd.yaml

Identify:

--data-dir
--listen-client-urls
--cert-file
--key-file
--trusted-ca-file

Create snapshot

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/etcd-backup.db

Validate snapshot

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Restore to a separate directory

sudo etcdutl snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore-from-backup

Configure etcd to use the restored directory

Update:

--data-dir=/var/lib/etcd-restore-from-backup

and the corresponding hostPath.

Restart/reconcile kubelet if required

sudo systemctl restart kubelet

Verify

k get pods -n kube-system

Look for:

etcd-osboxes   1/1   Running

40. What this lab actually taught me

The most valuable part of this exercise was not memorizing one command.

It was learning how the pieces connect:

Kubernetes
    |
    +-- kube-apiserver
    |
    +-- kube-controller-manager
    |
    +-- kube-scheduler
    |
    +-- kubelet
            |
            +-- static Pod manifests
                    |
                    +-- etcd
                            |
                            +-- /var/lib/etcd

The backup process interacted directly with etcd:

etcd
  |
  +-- TLS certificates
  |
  +-- etcdctl
  |
  +-- snapshot
  |
  +-- /opt/etcd-backup.db

The restore process then created:

/var/lib/etcd-restore-from-backup

and the static Pod was changed to consume that restored data.

Finally, kubelet reconciled the static Pod and etcd returned to:

1/1 Running

41. Final takeaway

An etcd backup is not just another Kubernetes YAML export.

The core workflow is:

1. Verify etcd is healthy.
2. Find the etcd endpoint and TLS certificates.
3. Take an etcd snapshot.
4. Validate the snapshot.
5. Restore the snapshot into a separate data directory.
6. Configure the etcd static Pod to use the restored data.
7. Let kubelet reconcile the static Pod.
8. Verify etcd and the Kubernetes control plane.

The terminal session also demonstrated something that documentation often hides: the commands do not always work on the first attempt.

There were:

spelling mistakes;
incorrect flags;
permission problems;
wrong filenames;
deprecated commands;
missing utilities;
namespace mistakes;
RBAC errors;
attempts to use sudo where it did not apply;
a static etcd Pod that remained Pending;
and finally a kubelet restart that brought etcd back to Running.

Those failures were not separate from the learning experience. They were the learning experience.

The most useful troubleshooting habit from this lab is simple:

When Kubernetes gives you an error, do not immediately change random things. Read the error, identify which layer produced it, inspect the relevant component, make the smallest reasonable correction, and verify the result.

That is the difference between memorizing Kubernetes commands and actually learning how the control plane works.

Terminal command reference from the lab

Cluster health

k get pods -A
k get pods -A -w
k get nodes
k get all
k get all -A

Static Pod manifests

cd /etc/kubernetes/manifests
ls
sudo cat etcd.yaml

Install tooling

sudo apt install etcd-client
sudo apt install etcd-server

Configure etcdctl

export ETCDCTL_API=3

Backup

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/etcd-backup.db

Validate

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Test workload

k create deploy nginx --image=nginx
k get po
k get po -w
k describe po nginx-7f8fbb96d-9f7wm
k get all

Delete workload

k delete deploy nginx

Restore

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore-from-backup

Troubleshoot etcd

k get po -n=kube-system
k describe po -n=kube-system etcd-osboxes
k get pod etcd-osboxes -n kube-system -o wide
k logs etcd-osboxes -n kube-system
sudo systemctl status kubelet --no-pager
sudo systemctl restart kubelet

Final verification

k get po -n=kube-system
k describe po -n=kube-system etcd-osboxes

Kubernetes ETCD Backup And Restore

Introduction

1. First: What is etcd?

2. Verify that the cluster is healthy first

Terminal

3. An important distinction: Kubernetes YAML backup vs etcd backup

4. Finding the etcd static pod

5. Another wrong assumption: trying to use Docker

6. Inspecting the etcd manifest

7. Install etcdctl

8. ETCDCTL_API=3

9. Creating the first etcd snapshot — and making a typo

10. The next error: permission denied

11. The successful etcd backup

12. Validating the snapshot

Lesson 1: The filename was wrong

Lesson 2: The command was deprecated

13. Why create an nginx Deployment?

14. Waiting for the nginx Pod

15. Take another backup after the nginx Deployment exists

16. Verify the state we want to recover

17. Delete nginx

18. First restore mistake: wrong snapshot filename

19. Restore the snapshot into a separate data directory

20. Why use a separate restore directory?

21. More path mistakes while inspecting the restored data

22. Editing the static etcd manifest

23. Why moving the manifest files caused disruption

24. Another small typo

25. The RBAC error after the manifest changes

26. etcd was now Pending

27. Trying to read etcd logs

28. Inspecting kubelet

29. The recovery

30. Confirming the restored etcd configuration

31. What happened to the nginx Deployment?

32. The complete clean backup command

Endpoint

CA certificate

Client certificate

Private key

Snapshot command

33. The clean snapshot validation command

34. The clean restore command

35. Important troubleshooting lessons

Error 1: Wrong directory

Fix

Error 2: Docker container not found

Lesson

Error 3: Permission denied reading etcd files

Fix

Error 4: Invalid etcdctl flag

Fix

Error 5: Private key permission denied

Fix

Error 6: Wrong snapshot filename

Lesson

Error 7: etcdctl snapshot status deprecated

Fix

Error 8: etcdutl not installed

Fix

Error 9: sudo cd

Lesson

Error 10: Pod name not found

Fix

Error 11: kubectl logs without a Pod

Fix

Error 12: kubectl Forbidden

Error 13: etcd remained Pending

36. Why static Pods matter in this exercise

37. A safer mental model for etcd restore

38. What I would do differently next time

1. Check the exact paths first

2. Inspect the manifest before changing it

3. Prefer the current etcd utilities

4. Avoid unnecessary movement of all manifests

5. Verify each stage

39. A compact recovery checklist

Identify etcd

Inspect the manifest

Error 7: `etcdctl snapshot status` deprecated

Error 8: `etcdutl` not installed

Error 9: `sudo cd`

Error 11: `kubectl logs` without a Pod

Error 12: `kubectl` Forbidden