How to manage persistent storage¶
Like many other databases, etcd stores its state on disk. In a default deployment (as described in Deploy etcd), the file system attached to charmed etcd will be removed when charmed etcd is removed. The content of the etcd database would then be lost.
To decouple the life cycle of the etcd database from the life cycle of its deployment, charmed etcd provides a feature called persistent storage. It allows to keep storage volumes around, even after a deployed etcd application or unit has been removed.
The use cases can be broken down into two groups:
reusing storage from previous units but within the same etcd cluster/database (see: Same cluster scenario) or
reusing storage from another etcd cluster/database (see: Different cluster scenario)
Charmed etcd uses two different storage volumes:
datacontaining the raw data files (the actual database) written and managed by etcdarchivefor temporarily storing a backup file before uploading it to object storage or when downloading it from object storagelogsfor log files written by etcd
The following document will explain how to reuse storage volumes in charmed etcd.
Prerequisites¶
The storage volumes that can be attached to charmed etcd depend on the storage providers available in the cloud where charmed etcd gets deployed. Please refer to the documentation about Juju storage for more information on possible providers.
Create a storage pool¶
In our case, as we deploy charmed etcd in a local LXD cloud, we use the lxd storage provider to create a storage pool
for etcd:
juju create-storage-pool etcd-storage lxd volume-type=standard
Make sure the storage pool has been created:
juju storage-pools
Example output:
Name Provider Attributes
etcd-storage lxd volume-type=standard
loop loop
lxd lxd
lxd-btrfs lxd driver=btrfs lxd-pool=juju-btrfs
lxd-zfs lxd driver=zfs lxd-pool=juju-zfs zfs.pool_name=juju-lxd
rootfs rootfs
tmpfs tmpfs
Details about how to manage storage pools with Juju can be found in this guide.
Deploy with persistent storage¶
The decision about using the persistent storage feature of charmed etcd has to be made at deploy time. In order to create
persistent storage volumes, add the --storage option and define your storage parameters.
This command will deploy a charmed etcd application with three units, create two storage volumes of 8 GB each, and attach them as data and archive volume mounts:
juju deploy charmed-etcd -n 3 --storage data=etcd-storage,8G,1 --storage archive=etcd-storage,8G,1
You can track the progress by executing:
juju status --storage --watch=1s
When the application is ready, juju status --storage will show something similar to the sample output below:
Model Controller Cloud/Region Version SLA Timestamp
etcd dev-controller localhost/localhost 3.6.0 unsupported 13:00:45Z
App Version Status Scale Charm Channel Rev Exposed Message
charmed-etcd active 3 charmed-etcd 0 no
Unit Workload Agent Machine Public address Ports Message
charmed-etcd/0* active idle 0 10.105.253.32
charmed-etcd/1 active idle 1 10.105.253.52
charmed-etcd/2 active idle 2 10.105.253.148
Machine State Address Inst id Base AZ Message
0 started 10.105.253.32 juju-84b57b-0 [email protected] Running
1 started 10.105.253.52 juju-84b57b-1 [email protected] Running
2 started 10.105.253.148 juju-84b57b-2 [email protected] Running
Storage Unit Storage ID Type Pool Mountpoint Size Status Message
charmed-etcd/0 archive/0 filesystem etcd-storage /var/snap/charmed-etcd/common/archive 8.0 GiB attached
charmed-etcd/0 data/1 filesystem etcd-storage /var/snap/charmed-etcd/common/var/lib/etcd 8.0 GiB attached
charmed-etcd/0 logs/2 filesystem rootfs /var/snap/charmed-etcd/common/var/log/etcd 76 GiB attached
charmed-etcd/1 archive/3 filesystem etcd-storage /var/snap/charmed-etcd/common/archive 8.0 GiB attached
charmed-etcd/1 data/4 filesystem etcd-storage /var/snap/charmed-etcd/common/var/lib/etcd 8.0 GiB attached
charmed-etcd/1 logs/5 filesystem rootfs /var/snap/charmed-etcd/common/var/log/etcd 76 GiB attached
charmed-etcd/1 archive/6 filesystem etcd-storage /var/snap/charmed-etcd/common/archive 8.0 GiB attached
charmed-etcd/2 data/7 filesystem etcd-storage /var/snap/charmed-etcd/common/var/lib/etcd 8.0 GiB attached
charmed-etcd/2 logs/8 filesystem rootfs /var/snap/charmed-etcd/common/var/log/etcd 76 GiB attached
As you can see, volumes from the etcd-storage pool have been attached as the data and archive volume mounts,
whereas for the logs volume mount, non-persistent storage from rootfs has been used. The logs storage will be
removed when the respective unit gets removed, while the data storage will persist.
Same cluster scenario¶
In this scenario, we want to reuse storage from previous units but within the same etcd cluster/database. This could be useful when you want to scale down your etcd database temporarily without completely removing it.
To scale down, run the juju remove-unit command:
juju remove-unit charmed-etcd/0 charmed-etcd/1 charmed-etcd/2
Example output:
WARNING This command will perform the following actions:
will remove unit charmed-etcd/0
- will remove storage logs/2
- will detach storage archive/0
- will detach storage data/1
will remove unit charmed-etcd/1
- will remove storage logs/5
- will detach storage archive/3
- will detach storage data/4
will remove unit charmed-etcd/2
- will remove storage logs/8
- will detach storage archive/6
- will detach storage data/7
Continue [y/N]? y
As you can see, the logs storage will be removed while the data storage will only be detached.
You can check this after removing the units by running the juju status --storage command:
Model Controller Cloud/Region Version SLA Timestamp
etcd dev-controller localhost/localhost 3.6.0 unsupported 13:02:34Z
App Version Status Scale Charm Channel Rev Exposed Message
charmed-etcd unknown 0 charmed-etcd 0 no
Storage Unit Storage ID Type Pool Mountpoint Size Status Message
archive/0 filesystem etcd-storage 8.0 GiB detached
archive/3 filesystem etcd-storage 8.0 GiB detached
archive/6 filesystem etcd-storage 8.0 GiB detached
data/1 filesystem etcd-storage 8.0 GiB detached
data/4 filesystem etcd-storage 8.0 GiB detached
data/7 filesystem etcd-storage 8.0 GiB detached
When you want to scale up your etcd database again, attach the desired data volume to the new unit of charmed etcd:
juju add-unit charmed-etcd --attach-storage=data/1 --attach-storage=archive/0
In order to scale up and resume your cluster with the most up-to-date state and data, make sure to attach the volume with most recent data (meaning: of the unit you removed last).
You can track the progress by running juju status --storage --watch=3s:
Model Controller Cloud/Region Version SLA Timestamp
etcd dev-controller localhost/localhost 3.6.0 unsupported 13:06:22Z
App Version Status Scale Charm Channel Rev Exposed Message
charmed-etcd active 1 charmed-etcd 0 no
Unit Workload Agent Machine Public address Ports Message
charmed-etcd/3* active idle 3 10.105.253.200
Machine State Address Inst id Base AZ Message
3 started 10.105.253.200 juju-84b57b-3 [email protected] Running
Storage Unit Storage ID Type Pool Mountpoint Size Status Message
archive/3 filesystem etcd-storage 8.0 GiB detached
archive/6 filesystem etcd-storage 8.0 GiB detached
data/4 filesystem etcd-storage 8.0 GiB detached
data/7 filesystem etcd-storage 8.0 GiB detached
charmed-etcd/3 archive/0 filesystem etcd-storage /var/snap/charmed-etcd/common/archive 8.0 GiB attached
charmed-etcd/3 data/1 filesystem etcd-storage /var/snap/charmed-etcd/common/var/lib/etcd 8.0 GiB attached
charmed-etcd/3 logs/9 filesystem rootfs /var/snap/charmed-etcd/common/var/log/etcd 76 GiB attached
As you can see, the new unit charmed-etcd/3 has been deployed with the existing archive/0 and data/1 volumes,
and the new volume logs/6 attached to it.
In order to scale up to three units again, add two more juju add-unit commands with the other storage volumes:
juju add-unit charmed-etcd --attach-storage=data/4 --attach-storage=archive/3
juju add-unit charmed-etcd --attach-storage=data/7 --attach-storage=archive/6
The data of these volumes will not be taken into account though, they will receive a full data transfer from the leader.
When the application is ready, you will see all three data volumes attached again:
juju status --storage --watch=3s
Example output:
Model Controller Cloud/Region Version SLA Timestamp
etcd dev-controller localhost/localhost 3.6.0 unsupported 13:09:33Z
App Version Status Scale Charm Channel Rev Exposed Message
charmed-etcd active 3 charmed-etcd 0 no
Unit Workload Agent Machine Public address Ports Message
charmed-etcd/3* active idle 3 10.105.253.200
charmed-etcd/4 active idle 4 10.105.253.33
charmed-etcd/5 active idle 5 10.105.253.6
Machine State Address Inst id Base AZ Message
3 started 10.105.253.200 juju-84b57b-3 [email protected] Running
4 started 10.105.253.33 juju-84b57b-4 [email protected] Running
5 started 10.105.253.6 juju-84b57b-5 [email protected] Running
Storage Unit Storage ID Type Pool Mountpoint Size Status Message
charmed-etcd/3 archive/0 filesystem etcd-storage /var/snap/charmed-etcd/common/archive 8.0 GiB attached
charmed-etcd/3 data/1 filesystem etcd-storage /var/snap/charmed-etcd/common/var/lib/etcd 8.0 GiB attached
charmed-etcd/3 logs/9 filesystem rootfs /var/snap/charmed-etcd/common/var/log/etcd 76 GiB attached
charmed-etcd/4 archive/3 filesystem etcd-storage /var/snap/charmed-etcd/common/archive 8.0 GiB attached
charmed-etcd/4 data/4 filesystem etcd-storage /var/snap/charmed-etcd/common/var/lib/etcd 8.0 GiB attached
charmed-etcd/4 logs/10 filesystem rootfs /var/snap/charmed-etcd/common/var/log/etcd 76 GiB attached
charmed-etcd/5 archive/6 filesystem etcd-storage /var/snap/charmed-etcd/common/archive 8.0 GiB attached
charmed-etcd/5 data/7 filesystem etcd-storage /var/snap/charmed-etcd/common/var/lib/etcd 8.0 GiB attached
charmed-etcd/5 logs/11 filesystem rootfs /var/snap/charmed-etcd/common/var/log/etcd 76 GiB attached
Different cluster scenario¶
In this scenario, we want to reuse existing storage from another etcd cluster/database.
Safe removal¶
Caution
Before you remove your etcd cluster, either:
save the credentials for the admin-user
or configure a user-defined password
By default, charmed etcd enables authentication. That means you can not access an existing etcd database without providing credentials. This is also valid for the admin-user charmed etcd uses to operate the cluster.
At any time before removing your existing etcd cluster, you can provide a user-defined password for the admin user and configure it to charmed etcd.
For instructions on how to configure credentials to etcd, refer to the guide How to manage passwords.
With a password now configured, it is safe to remove your existing charmed etcd application:
juju remove-application charmed-etcd
Example output:
WARNING This command will perform the following actions:
will remove application charmed-etcd
- will remove unit charmed-etcd/3
- will remove unit charmed-etcd/4
- will remove unit charmed-etcd/5
- will remove storage logs/9
- will remove storage logs/10
- will remove storage logs/11
- will detach storage archive/0
- will detach storage archive/3
- will detach storage archive/6
- will detach storage data/1
- will detach storage data/4
- will detach storage data/7
Continue [y/N]? y
You can check the status after removal by running the juju status --storage command:
Model Controller Cloud/Region Version SLA Timestamp
etcd dev-controller localhost/localhost 3.6.0 unsupported 13:11:11Z
Storage Unit Storage ID Type Pool Mountpoint Size Status Message
archive/0 filesystem etcd-storage 8.0 GiB detached
archive/3 filesystem etcd-storage 8.0 GiB detached
archive/6 filesystem etcd-storage 8.0 GiB detached
data/1 filesystem etcd-storage 8.0 GiB detached
data/4 filesystem etcd-storage 8.0 GiB detached
data/7 filesystem etcd-storage 8.0 GiB detached
Model "admin/etcd" is empty.
Deploy new cluster with existing storage¶
After you removed your charmed etcd application, deploy a new one attaching the data volume with most recent data and
configure the secret URI containing the admin user’s password.
In order to deploy your new cluster the most up-to-date state and data, make sure to attach the volume with most recent data (meaning: of the unit you removed last):
juju deploy charmed-etcd etcd --attach-storage=data/1 --attach-storage=archive/0 --config system-users=secret:cuvh9ggv7vbc46jefvjg
To be able to read the password from the secret, grant secret access to the new charmed etcd application again:
juju grant-secret mysecret charmed-etcd
Watch the progress of your deployment again with juju status --storage --watch=3s:
Model Controller Cloud/Region Version SLA Timestamp
etcd dev-controller localhost/localhost 3.6.0 unsupported 13:25:01Z
App Version Status Scale Charm Channel Rev Exposed Message
charmed-etcd active 1 charmed-etcd 3 no
Unit Workload Agent Machine Public address Ports Message
charmed-etcd/8* active idle 8 10.105.253.225
Machine State Address Inst id Base AZ Message
8 started 10.105.253.225 juju-84b57b-8 [email protected] Running
Storage Unit Storage ID Type Pool Mountpoint Size Status Message
archive/3 filesystem etcd-storage 8.0 GiB detached
archive/6 filesystem etcd-storage 8.0 GiB detached
data/4 filesystem etcd-storage 8.0 GiB detached
data/7 filesystem etcd-storage 8.0 GiB detached
charmed-etcd/8 archive/0 filesystem etcd-storage /var/snap/charmed-etcd/common/archive 8.0 GiB attached
charmed-etcd/8 data/1 filesystem etcd-storage /var/snap/charmed-etcd/common/var/lib/etcd 8.0 GiB attached
charmed-etcd/8 logs/12 filesystem rootfs /var/snap/charmed-etcd/common/var/log/etcd 76 GiB attached
Once the application is in active status, scale up attaching the other volumes:
juju add-unit charmed-etcd --attach-storage=data/4 --attach-storage=archive/3
juju add-unit charmed-etcd --attach-storage=data/7--attach-storage=archive/6
Again: The data of these volumes will not be taken into account, they will receive a full data transfer from the leader.
When the deployment is complete, your new charmed etcd application is available with previously used data:
juju status --storage --watch=3s
Example output:
Model Controller Cloud/Region Version SLA Timestamp
etcd dev-controller localhost/localhost 3.6.0 unsupported 13:29:48Z
App Version Status Scale Charm Channel Rev Exposed Message
charmed-etcd active 3 charmed-etcd 3 no
Unit Workload Agent Machine Public address Ports Message
charmed-etcd/8* active idle 8 10.105.253.225
charmed-etcd/9 active idle 9 10.105.253.154
charmed-etcd/10 active idle 10 10.105.253.220
Machine State Address Inst id Base AZ Message
8 started 10.105.253.225 juju-84b57b-8 [email protected] Running
9 started 10.105.253.154 juju-84b57b-9 [email protected] Running
10 started 10.105.253.220 juju-84b57b-10 [email protected] Running
Storage Unit Storage ID Type Pool Mountpoint Size Status Message
charmed-etcd/10 archive/6 filesystem etcd-storage /var/snap/charmed-etcd/common/archive 8.0 GiB attached
charmed-etcd/10 data/7 filesystem etcd-storage /var/snap/charmed-etcd/common/var/lib/etcd 8.0 GiB attached
charmed-etcd/10 logs/14 filesystem rootfs /var/snap/charmed-etcd/common/var/log/etcd 76 GiB attached
charmed-etcd/8 archive/0 filesystem etcd-storage /var/snap/charmed-etcd/common/archive 8.0 GiB attached
charmed-etcd/8 data/1 filesystem etcd-storage /var/snap/charmed-etcd/common/var/lib/etcd 8.0 GiB attached
charmed-etcd/8 logs/12 filesystem rootfs /var/snap/charmed-etcd/common/var/log/etcd 76 GiB attached
charmed-etcd/9 archive/3 filesystem etcd-storage /var/snap/charmed-etcd/common/archive 8.0 GiB attached
charmed-etcd/9 data/4 filesystem etcd-storage /var/snap/charmed-etcd/common/var/lib/etcd 8.0 GiB attached
charmed-etcd/9 logs/13 filesystem rootfs /var/snap/charmed-etcd/common/var/log/etcd 76 GiB attached
Remove persistent storage¶
To remove all remaining volumes and their data, use the --destroy-storage parameter when removing your application:
juju remove-application charmed-etcd --destroy-storage
Example output:
WARNING This command will perform the following actions:
will remove application charmed-etcd
- will remove unit charmed-etcd/8
- will remove unit charmed-etcd/9
- will remove unit charmed-etcd/10
- will remove storage archive/0
- will remove storage logs/12
- will remove storage data/1
- will remove storage archive/3
- will remove storage logs/13
- will remove storage data/4
- will remove storage archive/6
- will remove storage logs/14
- will remove storage data/7
Continue [y/N]? y
To ensure that the storage volumes were removed, run the juju storage command. The output will contain the message:
No storage to display.
To clean up the secret with the admin user credentials, obtain the secret ID:
juju secrets
Example output:
ID Name Owner Rotation Revision Last updated
cuvh9ggv7vbc46jefvjg mysecret <model> never 1 14 minutes ago
Then, run the command juju remove-secret <secret_id>:
juju remove-secret cuvh9ggv7vbc46jefvjg