# High Availability

> What to prepare for a three-machine Buttons HA cluster, and how the installation usually goes.

In a High Availability (HA) deployment, Buttons runs on three machines at once. If one machine fails or is taken down for maintenance, the other two keep Buttons running. Understand HA clustering explains how this works and why it takes exactly three machines. HA needs an Enterprise license. In most cases Bitfocus installs the cluster for you: you prepare the machines and the network details below, and Bitfocus does the rest. The installation steps further down show how an install usually goes, for when you do it yourself together with Bitfocus. What you need An Enterprise license. Once HA is on, a lower license tier counts as no license at all. Three machines running Ubuntu 24.04 LTS. Dedicated physical machines are best. Virtual machines also work if each one runs on a separate physical host with enough performance to spare. If two of them share a host, losing that host takes down two of the three machines, and Buttons stops. Enough hardware on each machine, as listed in the table below. Four IP addresses in the same subnet: a fixed address for each machine, and one free address for the virtual IP. You reach Buttons through the virtual IP. It always points at one of the three machines, and Buttons moves it to another machine if that one goes down. A user account on each machine that you can log in to over SSH and that can use sudo without typing a password. The installer runs its commands over SSH and can't answer a password prompt. Optionally, a DNS name such as buttons.example.com that points at the virtual IP. Set it up before you install if you want to use it. Each of the three machines needs at least this hardware: Hardware Minimum Recommended CPU cores 8 16 RAM 16 GB 16 GB or more Disk space for /var 250 GB 500 GB Install Buttons When you buy Buttons HA, your Bitfocus salesperson sends you two download links: one for the installation tool and one for the installation bundle. The commands below may change between releases, and Bitfocus can help if a step looks different. Log in to the first machine and run every step there. Download the installation tool. Replace <tool link> with the link for the installation tool. wget '<tool link>' -O - | gunzip -c > buttons-ctl chmod +x buttons-ctl Download and extract the installation bundle. Replace <bundle link> with the link for the bundle. The bundle contains everything the three machines need, so they don't have to download anything else during the installation. The files end up in a folder called airgap-bundle . wget -q '<bundle link>' -O - | tar zxvf - Find the name of the network interface. Run ip address show and look for the block that lists the machine's IP address. The interface name is at the start of that block, for example eth0 or a name starting with en , such as ens18 . Start the configuration wizard: ./buttons-ctl config wizard Answer the wizard's questions like this: Cluster name : keep the default. IP address , SSH username and main network interface for VM 1, 2 and 3: each machine's IP address, the user you log in with, and the interface name from step 3. SSH key : if you already have an SSH key that logs in to all three machines, answer yes and select it. If not, answer no and let the wizard generate one. Virtual IP : the free address for the virtual IP. It must be in the same subnet as the machines. Virtual IP interface : the same interface name as before. Domain name : the DNS name that points at the virtual IP. If you don't have one, enter the virtual IP. Check the summary and select Save & Write Config . The wizard saves the configuration as config.yaml in the current folder. If the wizard generated a new SSH key, copy it to the three machines. You're asked for each machine's password once. ./buttons-ctl config push-keys -c config.yaml Check that the machines are ready: ./buttons-ctl config precheck The precheck logs in to each machine and marks each check pass , warn or fail . Fix anything marked fail before you continue. The installation fixes most warnings by itself, such as missing packages or time synchronization being off. Install Buttons: ./buttons-ctl install -c config.yaml -b airgap-bundle --airgapped --auto The installer runs the precheck again, fixes what it can, and then installs Buttons on all three machines. This takes a while, and it shows what it's doing as it goes. When the installation is done, open https:// followed by the virtual IP or the DNS name in Chrome, for example https://buttons.example.com . Buttons uses a self-signed certificate, so the browser warns that the connection isn't private the first time. Accept the warning to continue. Buttons then opens the First-time setup page, where you create the first administrator. See Create the first administrator . The self-signed certificate is enough in most cases. If you use a DNS name and want a trusted certificate for it, upload one under Settings after you sign in. See Replace the HTTPS certificate . Forward incoming traffic to connections and workflows Connect to Buttons only through the virtual IP. A connection or workflow that other devices connect in to, for example a module that listens on a return port or a workflow with a REST Server node, also needs a port forwarding rule so the virtual IP forwards that port to it. See: Forward incoming traffic to a connection in an HA deployment Forward incoming traffic to a workflow in an HA deployment Tally Server (Listen) connections don't need a rule: Buttons forwards their ports automatically. Health and failover The System Health banner and the Services page show the state of every service on every machine. See Interpret system health and restart services safely . To take a machine down for maintenance, use Cordon & Drain on the Services page instead of switching it off. Take down one machine at a time, and wait until it's back before you start on the next. Buttons keeps running on two machines, but not on one. If you get stuck What you see What to try The precheck fails on RAM or disk space. Give the machine more memory or a larger disk, then run the precheck again. The precheck fails because an interface is missing. Check the interface name with ip address show on that machine, and run the wizard again with the right name. The precheck or installer can't log in to a machine. Make sure the SSH key is on that machine (run config push-keys again) and that the user can run sudo without a password. Buttons shows no active license, or licensed features are missing. HA needs an Enterprise license. Any lower tier counts as no license at all. Where to go next Understand HA clustering Choose an installation path Interpret system health and restart services safely Replace the HTTPS certificate Back up and recover a High Availability deployment Manage your Buttons license