Migrating GitLab by using direct transfer

DETAILS: Tier: Free, Premium, Ultimate Offering: GitLab.com, Self-managed, GitLab Dedicated

• Enabled on GitLab.com in GitLab 15.6.
• New application setting bulk_import_enabled introduced in GitLab 15.8. bulk_import feature flag removed.
• bulk_import_projects feature flag removed in GitLab 15.10.

You can migrate GitLab groups:

• From self-managed GitLab to GitLab.com.
• From GitLab.com to self-managed GitLab.
• From one self-managed GitLab instance to another.
• Between groups in the same GitLab instance.

Migration by direct transfer creates a new copy of the group. If you want to move groups instead of copying groups, you can transfer groups if the groups are in the same GitLab instance. Transferring groups instead of migrating them is a faster and more complete option.

You can migrate groups in two ways:

• By direct transfer (recommended).
• By uploading an export file.

If you migrate from GitLab.com to self-managed GitLab, an administrator can create users on the self-managed GitLab instance.

On self-managed GitLab, by default migrating group items is not available. To show the feature, an administrator can enable it in application settings.

Migrating groups by direct transfer copies the groups from one place to another. You can:

• Copy many groups at once.
• In the GitLab UI, copy top-level groups to:
  • Another top-level group.
  • The subgroup of any existing top-level group.
  • Another GitLab instance, including GitLab.com.
• In the API, copy top-level groups and subgroups to these locations.
• Copy groups with projects (in beta and not ready for production use) or without projects. Copying projects with groups is available:
  • On GitLab.com by default.

Not all group and project resources are copied. See list of copied resources below:

• Migrated group items.
• Migrated project items.

WARNING: Importing groups with projects is in beta. This feature is not ready for production use.

We invite you to leave your feedback about migrating by direct transfer in the feedback issue.

Migrating specific projects

Migrating groups by using direct transfer in the GitLab UI migrates all projects in the group. If you want to migrate only specific projects in the group by using direct transfer, you must use the API.

Known issues

• Because of issue 406685, files with a filename longer than 255 characters are not migrated.
• In GitLab 16.1 and earlier, you should not use direct transfer with scheduled scan execution policies.
• For a list of other known issues, see epic 6629.
• In GitLab 16.9 and earlier, because of issue 438422, you might see the DiffNote::NoteDiffFileCreationError error. When this error occurs, the diff of a note on a merge request's diff is missing, but the note and the merge request are still imported.
• When imported from the source instance, shared members are created as direct members on the destination unless those memberships already exist on the destination. This means that importing a top-level group on the source instance to a top-level group on the destination instance always creates direct members in projects, even though the source top-level group contains the necessary shared membership hierarchy details. Support for full replication of shared memberships is proposed in issue 458345.

Estimating migration duration

Estimating the duration of migration by direct transfer is difficult. The following factors affect migration duration:

• Hardware and database resources available on the source and destination GitLab instances. More resources on the source and destination instances can result in shorter migration duration because:
  • The source instance receives API requests, and extracts and serializes the entities to export.
  • The destination instance runs the jobs and creates the entities in its database.
• Complexity and size of data to be exported. For example, imagine you want to migrate two different projects with 1000 merge requests each. The two projects can take very different amounts of time to migrate if one of the projects has a lot more attachments, comments, and other items on the merge requests. Therefore, the number of merge requests on a project is a poor predictor of how long a project will take to migrate.

There's no exact formula to reliably estimate a migration. However, the average durations of each pipeline worker importing a project relation can help you to get an idea of how long importing your projects might take:

Though it's difficult to predict migration duration, we've seen:

• 100 projects (19.9k issues, 83k merge requests, 100k+ pipelines) migrated in 8 hours.
• 1926 projects (22k issues, 160k merge requests, 1.1 million pipelines) migrated in 34 hours.

If you are migrating large projects and encounter problems with timeouts or duration of the migration, see Reducing migration duration.

Reducing migration duration

These are some strategies for reducing the duration of migrations that use direct transfer.

Add Sidekiq workers to the destination instance

A single direct transfer migration runs 5 entities (groups or projects) per import at a time, independent of the number of workers available on the destination instance. That said, adding more Sidekiq worker processes on the destination instance speeds up migration by decreasing the time it takes to import each entity.

To add more Sidekiq workers on the destination instance, you can either:

1. Use routing rules. This approach creates additional Sidekiq workers that are dedicated to direct transfer operations.
2. Start multiple Sidekiq processes. This approach allows all queues in GitLab to make use of the additional Sidekiq worker processes.

An example of how to use the routing rules is below. This example:

• Can be added to the /etc/gitlab/gitlab.rb file on a destination instance, with a subsequent reconfigure to apply the changes.
• Creates four Sidekiq worker processes. Three of the processes are used exclusively for the direct transfer importer queues, and the last process is used for all other queues.
• Should have a number of processes set that, at most, equal (and not exceed) the number of CPU cores you want to dedicate to Sidekiq. The Sidekiq worker process uses no more than one CPU core.

sidekiq['routing_rules'] = [
  ['feature_category=importers', 'importers'],
  ['*', 'default']
]

sidekiq['queue_groups'] = [
  # Run two processes just for importers
  'importers',
  'importers',
  'importers',

  # Run one 'catchall' process on the default and mailers queues
  'default,mailers'
]

If you are using GitLab 16.11 and earlier, explicitly disable any queue selectors:

sidekiq['queue_selector'] = false

Increasing the number of workers on the destination instance helps reduce the migration duration until the source instance hardware resources are saturated. Exporting and importing relations in batches, available by default from GitLab 16.8, makes having enough available workers on the destination instance even more useful.

Redistribute large projects or start separate migrations

The number of workers on the source instance should be enough to export the 5 concurrent entities in parallel (for each running import). Otherwise, there can be delays and potential timeouts as the destination is waiting for exported data to become available.

Distributing projects in different groups helps to avoid timeouts. If several large projects are in the same group, you can:

1. Move large projects to different groups or subgroups.
2. Start separate migrations each group and subgroup.

The GitLab UI can only migrate top-level groups. Using the API, you can also migrate subgroups.

Limits

• Eight hour time limit on migrations removed in GitLab 16.7.

Hardcoded limits apply on migration by direct transfer.

Configurable limits are also available.

In GitLab 16.3 and later, the following previously hard-coded settings are configurable:

• Maximum relation size that can be downloaded from the source instance (set to 5 GiB).
• Maximum size of a decompressed archive (set to 10 GiB).

You can test the maximum relation size limit using these APIs:

• Group relations export API.
• Project relations export API

If either API produces files larger than the maximum relation size limit, group migration by direct transfer fails.

Visibility rules

After migration:

• Private groups and projects stay private.
• Internal groups and projects:
  • Stay internal when copied into an internal group unless internal visibility is restricted. In that case, the groups and projects become private.
  • Become private when copied into a private group.
• Public groups and projects:
  • Stay public when copied into a public group unless public visibility is restricted. In that case, the groups and projects become internal.
  • Become internal when copied into an internal group unless internal visibility is restricted. In that case, the groups and projects become private.
  • Become private when copied into a private group.

If you used a private network on your source instance to hide content from the general public, make sure to have a similar setup on the destination instance, or to import into a private group.

Memberships

• Importing of shared and inherited shared members was introduced in GitLab 16.3.
• Introduced in GitLab 16.11, shared and inherited shared members are no longer imported as direct members if they are already shared or inherited shared members of the imported group or project.

Group and project members are imported if the user account prerequisites are followed.

All direct and indirect members are imported.

Indirect members are imported as direct members if:

• They are not already an indirect member of the target namespace.
• They are an indirect member, but have a lower permission.

There is a known issue affecting the transfer of shared memberships.

Migration by direct transfer process

See Migrate groups and projects by using direct transfer.