Upgrading the Geo sites

DETAILS: Tier: Premium, Ultimate Offering: Self-managed

WARNING: Read these sections carefully before updating your Geo sites. Not following version-specific upgrade steps may result in unexpected downtime. If you have any specific questions, contact Support. A database major version upgrade requires re-initializing the PostgreSQL replication to Geo secondaries. This applies to both Linux-packaged and externally-managed databases. This may result in a larger than expected downtime.

Upgrading Geo sites involves performing:

1. Version-specific upgrade steps, depending on the version being upgraded to or from.
2. General upgrade steps, for all upgrades.

General upgrade steps

NOTE: These general upgrade steps require downtime in a multi-node setup. If you want to avoid downtime, consider using zero-downtime upgrades.

To upgrade the Geo sites when a new GitLab version is released, upgrade primary and all secondary sites:

1. Optional. Pause replication on each secondary site to protect the disaster recovery (DR) capability of the secondary sites.

2. SSH into each node of the primary site.

3. Upgrade GitLab on the primary site.

4. Perform testing on the primary site, particularly if you paused replication in step 1 to protect DR. There are some suggestions for post-upgrade testing in the upgrade documentation.

5. Ensure that the secrets in the /etc/gitlab/gitlab-secrets.json file of both the primary site and the secondary site are the same. The file must be the same on all of a site's nodes.

6. SSH into each node of secondary sites.

7. Upgrade GitLab on each secondary site.

8. If you paused replication in step 1, resume replication on each secondary. Then, restart Puma and Sidekiq on each secondary site. This is to ensure they are initialized against the newer database schema that is now replicated from the previously upgraded primary site.

   sudo gitlab-ctl restart sidekiq
   sudo gitlab-ctl restart puma

9. Test primary and secondary sites, and check version in each.

Check status after upgrading

Now that the upgrade process is complete, you may want to check whether everything is working correctly:

1. Run the Geo Rake task on an application node for the primary and secondary sites. Everything should be green:

   sudo gitlab-rake gitlab:geo:check

2. Check the primary site's Geo dashboard for any errors.

3. Test the data replication by pushing code to the primary site and see if it is received by secondary sites.

If you encounter any issues, see the Geo troubleshooting guide.