Code intelligence development guidelines

This document describes the design behind Code Intelligence.

The built-in Code Intelligence in GitLab is powered by LSIF and comes down to generating an LSIF document for a project in a CI job, processing the data, uploading it as a CI artifact and displaying this information for the files in the project.

Here is a sequence diagram for uploading an LSIF artifact:

%%{init: { "fontFamily": "GitLab Sans" }}%%
sequenceDiagram
    accTitle: Sequence diagram for LSIF artifact uploads
    accDescr: The process of how Runner, Workhorse, Rails, and object storage work together to upload an artifact.

    participant Runner
    participant Workhorse
    participant Rails
    participant Object Storage

    Runner->>+Workhorse: POST /v4/jobs/:id/artifacts
    Workhorse->>+Rails: POST /:id/artifacts/authorize
    Rails-->>-Workhorse: Respond with ProcessLsif header
    Note right of Workhorse: Process LSIF file
    Workhorse->>+Object Storage: Put file
    Object Storage-->>-Workhorse: request results
    Workhorse->>+Rails: POST /:id/artifacts
    Rails-->>-Workhorse: request results
    Workhorse-->>-Runner: request results

The CI/CD job generates a document in an LSIF format (usually dump.lsif) using an indexer for the language of a project. The format describes interactions between a method or function and its definitions or references. The document is marked to be stored as an LSIF report artifact.
After receiving a request for storing the artifact, Workhorse asks GitLab Rails to authorize the upload.
GitLab Rails validates whether the artifact can be uploaded and sends ProcessLsif: true header if the LSIF artifact can be processed.
Workhorse reads the LSIF document line by line and generates code intelligence data for each file in the project. The output is a zipped directory of JSON files which imitates the structure of the project:

Project:
```
app
  controllers
    application_controller.rb
  models
    application.rb
```
Generated data:
```
app
  controllers
    application_controller.rb.json
  models
    application.rb.json
```
The zipped directory is stored as a ZIP artifact. Workhorse replaces the original LSIF document with a set of JSON files in the ZIP artifact and generates metadata for it. The metadata makes it possible to view a single file in a ZIP file without unpacking or loading the whole file. That allows us to access code intelligence data for a single file.

When a file is viewed in the GitLab application, frontend fetches code intelligence data for the file directly from the object storage. The file contains information about code units in the file. For example:

[
    {
     "definition_path": "cmd/check/main.go#L4",
     "hover": [
         {
             "language": "go",
             "tokens": [
                 [
                     {
                         "class": "kn",
                         "value": "package"
                     },
                     {
                         "value": " "
                     },
                     {
                         "class": "s",
                         "value": "\"fmt\""
                     }
                 ]
             ]
         },
         {
             "value": "Package fmt implements formatted I/O with functions analogous to C's printf and scanf.  The format 'verbs' are derived from C's but are simpler. \n\n### hdr-PrintingPrinting\nThe verbs: \n\nGeneral: \n\n```\n%v\tthe value in a default format\n\twhen printing st..."
         }
     ],
     "start_char": 2,
     "start_line": 33
   }
   ...
 ]