Add support for distributed notes updates - #1473

To handle the situation when concurrent GitGitGadget operations want to
modify the state that is persisted in Git notes (such as
`refs/notes/gitgitgadget`), we need a way to reconcile diverging Git
notes.

Happily, the way GitGitGadget works makes this relatively
straight-forward, as concurrent operations usually do not conflict and
merely need to add/modify/remove independent attributes and/or values.
This allows us to treat these Git notes as kind of a Conflict-free
Replicated Data Type (https://en.wikipedia.org/wiki/CRDT).

The strategy implemented in this patch is relatively simple, and follows
the "local-first" idea (where state is applied locally first and only
then persisted remotely): The newly added method determines what changes
were made locally, and then reapplies them on top of the upstream note
commit. The local operations need to fall into one of the following
categories:

- a note has been appended to

- a single line of text (which may, or may not, be a stringified JSON
  object) is added as a note on an object that had not had a note yet

- a note containing a single line of text has been replaced with a new,
  single line of text

- a note that contained a JSON object (stringified as a single line of
  text) has been modified. Valid modifications include:

  - an attribute that was added

  - an attribute that has been removed

  - an attribute containing a primitive type (string, number or boolean)
    has been modified (if local and upstream diverged, local wins)

  - items from an attribute containing an array have been removed and/or
    added

  - an attribute containing a Plain Old Javascript Object has seen any
    of these modifications

Other modifications are as yet unhandled.

For historical reasons (*cough* Git commands implemented in Unix shell
scripts *cough*) it is impossible to add a Git note that does not end in
a newline character. Let's ensure that we don't do that, either.

To avoid tampering with a Git index, let's use a temporary index. This
requires the environment variable `GIT_INDEX_FILE` to be passed to the
Git processes, via the `env` attribute of the `IGitOptions` (which only
need to be declared and no other changes are needed because the
`options` are then used as dugite's `IGitExecutionOptions`, which
already has the `env` attribute).

Signed-off-by: Johannes Schindelin <[email protected]>

We actually do want to work exclusively on bare repositories. A recent
Git version requires either the `safe.bareRepository` config setting to
indicate that it is safe, or the git dir to be specified explicitly via
`--git-dir`. Let's do the latter.

This will become very important when we switch to using GitHub Actions
with ephemeral (and partial) clones of `gitgitgadget/git`.

Signed-off-by: Johannes Schindelin <[email protected]>

In the olden days, when GitGitGadget was still `mail-patch-series.sh`,
the source of truth was the local repository where it was run, and as
such, the branch needed to be pushed to the remote to make it, well,
public.

However, in the current setup, the source of truth is the
gitgitgadget/git (or git/git) repository's Pull Request, i.e. the branch
is already published.

Therefore, this branch publishing logic should have been removed in
6d397b0 (Remove support for the CLI mode, 2023-12-31)... better late
than never...

Signed-off-by: Johannes Schindelin <[email protected]>

Let's only ever push `refs/notes/gitgitgadget` from
`GitGitGadget.pushNotesRef()`. This will make the next change easier,
where we make notes pushing more resilient against concurrency by
incorporating the shiny new `GitNotes.notesSync()` call.

Signed-off-by: Johannes Schindelin <[email protected]>

GitGitGadget's state lives in a JSON object at the tip of the
`refs/notes/gitgitgadget` ref at https://github.com/gitgitgadget/git.

When GitGitGadget's automation runs, it typically modifies that, and
then pushes the new state.

In case of concurrent runs, this would cause a problem (which is why we
currently prevent that). But not for long! This here commit uses the
`notesSync()` function that was introduced _just_ for the purpose of
allowing concurrent runs, by merging whatever changed upstream and then
pushing again.

To allow for multiple concurrent contenders, let's try that a couple of
times before giving up, waiting for an increasingly long time.

This functionality is now made available via the shiny new
`GitNotes.push(url)` method.

This method will henceforth be used by the `GitGitGadget.pushNotesRef()`
method to guarantee robustness for concurrent GitGitGadget workflows.

Signed-off-by: Johannes Schindelin <[email protected]>

I'd like the following `misc-helper` commands to call the new
`GitNotes.push()` method as needed, to benefit from the just-introduced
support for concurrent GitGitGadget workflow runs:

- update-open-prs

- update-commit-mappings

- handle-open-prs

- handle-new-mails

Currently, these are handled in the "Update GitGitGadget's PRs" Pipeline at
https://dev.azure.com/gitgitgadget/git/_build/index?definitionId=2&_a=completed
and in the "Mirror Git List to GitGitGadget's PRs" Azure Pipeline at
https://dev.azure.com/gitgitgadget/git/_build/index?definitionId=5&_a=completed

The Pipeline definitions perform those pushes manually, and changing
CIHelper to do the push preemptively should allow for those definitions
to remain unchanged until the point in time when they are replaced by
workflows in https://github.com/gitgitgadget/gitgitgadget-workflows.

Technically, this commit does not even touch `misc-helper.ts`; The
reason is that the logic is actually implemented in the `CIHelper`
class, which is a good thing, because that class will become the
principal entry-point of above-mentioned GitHub workflows.

Also note that there is a slight change of (`push`) behavior in
`handle-open-prs`: This command calls `CIHelper.handleOpenPRs()`, which
in turn calls `CIHelper.handlePR()` in a loop. Now, the notes ref will
be pushed every time the latter updates something, where before, the
notes ref was only pushed once, at the very end. This is arguably safer
because PRs that have already been handled in a Pipeline run that fails
somewhere in the middle won't be handled again upon the re-run.

Eventually I'd like the same graceful notes ref pushing even for
script/update-mail-to-commit-notes.sh (which updates not the
`gitgitgadget`, but the `mail-to-commit` and `commit-to-mail` notes
refs), so that the "Update GitGitGadget's commit to mail notes" Pipeline at
https://dev.azure.com/gitgitgadget/git/_build/index?definitionId=9&_a=completed
can also be migrated to a GitHub workflow.

Signed-off-by: Johannes Schindelin <[email protected]>

While we may want to consider taking out the dry-run mode (as it is not
surfaced in GitGitGadget ever since 6d397b0 (Remove support for the CLI
mode, 2023-12-31) removed the ability to run it in command-line mode),
the mode still exists and pushing a tag (and now also notes ref) should
definitely not be done in dry-run mode.

Pointed-out-by: Chris. Webster <[email protected]>
Signed-off-by: Johannes Schindelin <[email protected]>

All current callers pass in a valid `options` parameter, therefore there
is no need any longer for the original logic to potentially initialize
those options.

Signed-off-by: Johannes Schindelin <[email protected]>

The `CIHelper.handlePR()` method already took care of updating the
notes, no need to do it again.

Signed-off-by: Johannes Schindelin <[email protected]>

… `handlePR()`

We already push the notes ref when updating the PR metadata (i.e. the
object attached to the blob whose contents are the PR URL), therefore it
makes only sense to combine that with any update to the
GitGitGadgetOptions (i.e. the object attached to the empty object).

This simplifies the logic (and the separation of concerns) quite a bit.

Suggested-by: Chris. Webster <[email protected]>
Signed-off-by: Johannes Schindelin <[email protected]>