This article details the procedure to shut down and restart HA servers (including the Applications and the DB) without causing any damage to the data and to the ongoing work.

If the primary server goes down, the system will send a notification to let you know.

Prerequisites

  • Master APP server
  • Slave APP server
  • Master DB server
  • Slave DB server

Shut down HA App Servers with “Graceful shutdown”

A “graceful shutdown” involves shutting down each server separately as shown in the following procedure.

  1. Connect via SSH to the applications – app1 + app2
  2. Run the following command on the application you want to shutdown:
    pcs cluster stop [node] → pcs cluster stop app1
  3. Verify that the system is still working on the other node (App2) by entering:
    pcs status

  1. After maintenance work is carried out on this application you can turn it back by entering:
    pcs cluster start [node]--> pcs cluster start app1
  2. Repeat the same steps for the other application.

Shut down HA App Servers with “Kill shutdown”

A kill shutdown involves shutting both applications down at the same time.

  1. Connect via SSH to the applications – app1 + app2
  2. Run the following command on both applications:
    pcs cluster stop --all
  3. After maintenance is carried out, restart both application nodes by entering the following command on both nodes:
    pcs cluster start --all
  4. Verify using pcs_status on one of the nodes that everything is back to normal and that all services are working.

Shut down HA DB Servers with Full Shutdown (both DB and Apps)

A full shutdown of the HA Servers involves shutting down the DBs and the applications using the procedure below.

An HA cluster DB will have both the primary and standby DB nodes running as shown in the picture below.

  1. If you want to maintain both Databases in parallel before they start, you need to shut down both applications using pcs cluster stop --all
  2. Connect to both DBs via SSH.
  3. Stop postgres services on both nodes ( N1+N2) by entering:
    systemctl stop postgresql-10
  4. After you finish maintenance, begin postgres services on N1
    systemctl start postgresql-10
  5. Rejoin N2 as standby node by entering:
    sudo su -
systemctl stop postgresql-10
su  postgres
cd ~
PGPASSWORD=<postgres user password> /usr/pgsql-10/bin/repmgr -f /etc/repmgr/10/repmgr.conf -h <N1 ip> -U repuser -d postgres standby clone --force-rewind --force
exit 
systemctl start postgresql-10
su postgres
/usr/pgsql-10/bin/repmgr standby register --force
  6. On both nodes, verify the DB has returned to the starting point with the following command:
    /usr/pgsql-10/bin/repmgr -f /etc/repmgr/10/repmgr.conf cluster show and verify as in the picture below
  7. Run the following command to restart the services on any of the two application nodes:
    pcs cluster start --all
  8. Verify on one of the nodes that everything is back to normal and all services are working:
    pcs_status

Shut down HA DB Servers with Full Shutdown (DB only)

This full shutdown procedure of the HA Servers involves shutting down just DBs but leaving the applications running.

Shut down Replica Node N2

  1. Connect with SSH
  2. Run the command in order to stop the postgres services:
    systemctl stop postgresql-10
  3. Verify that applications are working as usual.
  4. When maintenance work is done, restart the postgres services with
    systemctl start postgresql-10
  5. Wait for the N2 to replicate N1 to cover for loss of data before shutting down N1

Shutdown Node N1

  1. Connect to N1 and N2 with SSH
  2. Stop postgres services on N1 using the following command:
    systemctl stop postgresql-10
  3. On N2 switch to postgres user:
    su - postgres
  4. Type the command
    /usr/pgsql-10/bin/repmgr -f /etc/repmgr/10/repmgr.conf cluster show
  5. When N2 becomes the Primary server you can perform maintenance on N1. You should see the HA as failed as in the screenshot below:

Register N1 as standby

  1. In order for N1 to replicate the data created on N2, during maintenance of N1, enter the following:
    sudo su -
systemctl stop postgresql-10
su  postgres
cd ~
PGPASSWORD=<postgres user password> /usr/pgsql-10/bin/repmgr -f /etc/repmgr/10/repmgr.conf -h <N2 ip> -U repuser -d postgres standby clone --force-rewind --force
exit 
systemctl start postgresql-10
su postgres
/usr/pgsql-10/bin/repmgr standby register --force

Return to Starting Position

  1. On Node 2 (currently acting as Primary) stop postgres services:
    systemctl stop postgresql-10
  2. Go to N1 and wait for it to become primary:
    sudo - postgres
    /usr/pgsql-10/bin/repmgr -f /etc/repmgr/10/repmgr.conf cluster show
  3. Once N1 becomes primary, move to N2 and register N2 as standby:
    sudo su -
su  postgres
cd ~
PGPASSWORD=<postgres user password> /usr/pgsql-10/bin/repmgr -f /etc/repmgr/10/repmgr.conf -h <N1 ip> -U repuser -d postgres standby clone --force-rewind --force
exit 
systemctl start postgresql-10
su postgres
/usr/pgsql-10/bin/repmgr standby register --force
  4. Verify on both nodes that the cluster is back to the starting position
    /usr/pgsql-10/bin/repmgr -f /etc/repmgr/10/repmgr.conf cluster show
    and as shown in the screenshot below:
  5. Verify repmgr services are active on both nodes
    systemctl status repmgr10