# HA backup and recovery

> Where backups are kept in an HA deployment, how to keep a copy outside the cluster, and how to recover.

An HA deployment keeps running when one of its three machines fails, because every part of Buttons, including the database, runs on all three. That protects you against losing a machine. It doesn't protect you against a bad change, an accidental deletion or losing the whole cluster: a mistake is copied to all three machines just as quickly as everything else. For those cases you need a backup. This page covers only what's different in an HA deployment. Backups themselves work the same way as on a single machine: see Configure and monitor scheduled backups , Restore a backup and verify it and Export or import Buttons configuration . Where backups are kept Scheduled and manual backups are stored on storage that is copied to all three machines. A backup therefore survives losing any one machine, and you can restore it from Settings as usual. The backups are still kept inside the cluster, though. If you lose all three machines, or their storage, the backups are lost with them. Keep a copy outside the cluster To be able to recover from losing the whole cluster, keep a recent copy of your configuration somewhere else: Export the configuration from Settings → Configuration data , as described in Export configuration . Store the downloaded archive outside the cluster, for example on a file server or in your organization's backup system. Repeat this regularly, and after any large change. Note A future version of Buttons will be able to copy backups automatically to S3 or Azure Blob Storage. Until then, exporting the configuration is how you get a copy out of the cluster. Recover What to do depends on what you've lost: What happened What to do One machine failed. Nothing needs restoring. Buttons keeps running on the other two machines. Bring the failed machine back as soon as you can, because the cluster can't survive losing a second one. A change or deletion went wrong. Restore a backup from before the change, as described in Restore a backup and verify it . The whole cluster or its storage is lost. Contact Bitfocus to reinstall the cluster, then import your most recent exported archive, as described in Import configuration . A restore replaces the current configuration. Everyone connected is disconnected while it runs, and anything changed after the backup or export was made is lost. After a restore, give Buttons a moment to start all its services again on all three machines. Check that the System Health banner is back to OK before you rely on the result. See Interpret system health and restart services safely . Where to go next Deploy Buttons with High Availability Configure and monitor scheduled backups Restore a backup and verify it Export or import Buttons configuration Interpret system health and restart services safely