Improve GitHub bridge - #760

Hmmm, I'm not sure this fully makes sense.

A bridge should probably not write it's own file. I understand you are doing that to keep track of the import progress? There is already a restart mechanism, although just storing the timestamp of the last successfully imported issue: https://github.com/MichaelMure/git-bug/blob/1939949fcdd5d540fd20ead79b9c5d27e85e8f9d/bridge/core/bridge.go#L378-L382

Would it be reasonable to piggyback on this mechanism or to extend it if a more granular progress tracking is necessary?

Thank you for working on that.

I understand you are doing that to keep track of the import progress?

I used the file to keep track of the progress and then restart the import from where it has left off.

A bridge should probably not write it's own file.

Sure, the bridge should not write its own file. The solution with the file should be viewed as a proof of concept.

The GitHub API is more fragile these days. It's seems to me that we should have a good story to restart/continue an import, or to import only specific issues.

Would it be reasonable to piggyback on this mechanism or to extend it if a more granular progress tracking is necessary?

nixpkgs has about 24000 issues. Tracking the issue numbers would mean to just write the failed issue numbers somewhere (into the local config). I am not sure if that makes sense. The list could be quite long. Is the local config supposed to be also edited by humans?

What do you think about having a command line argument for importing particular issues? Then I could imagine having a simple message at the end of the import. Imagine something like the following.

$ git-bug bridge pull
...
new issue: e0612737cebde57cb49b3f596ce045235a260258f70037d0f9860759ceba3e04 
changed label: 682fcc468648af894d56930f714d36124db5ad5348e250ddb6ad42ce1b5b7bd0
import error: couldn't import issues 1254,174098,47786,50032
imported 24645 issues and 6644 identities with default bridge

$ git-bug bridge pull --issues 1254,174098,47786,50032
...
new issue: 3719580985bde516469b3f596ce045235a260258f70037d0f451451435b00002
import error: couldn't import issues 47786
imported 3 issues and 1 identities with default bridge

nixpkgs has about 24000 issues. Tracking the issue numbers would mean to just write the failed issue numbers somewhere (into the local config). I am not sure if that makes sense. The list could be quite long. Is the local config supposed to be also edited by humans?

The local config is not supposed to be edited by humans.

I was thinking, in addition (or even replacement) of git-bug.bridge.%s.lastImportTime (that record up to what time the importer succeeded), maybe we could have something like git-bug.bridge.%s.importStatus, which would hold an opaque string. This string would be up for each bridge to store whatever they want in to track the import progress and continue later. Typically something like base64(struct{IssueId string, CommentId string}{"1", "2"}). That way, bridge are free to do what make sense for them, while we still have a common process and code to drive them.

Would that make sense?

What do you think about having a command line argument for importing particular issues? Then I could imagine having a simple message at the end of the import. Imagine something like the following.

Yeah, importing a subset of issue is definitely something useful. The tricky part is to figure out a way to make that generic for all bridges.

In #241, there was a discussion about reusing git-bug's query language as such a bridge neutral way to define a selection. Each bridge would then apply the parsed query into their own filtering, either directly when querying the remote tracker or filtering before actually importing.

That said, this is a bit different than importing specific issue number, but we could extend the query language to support that, it kinda make sense.

To be clear about the last part, it could also make sense to have two different ways to configure the import: one for specific issues, one with a query. I'm just thinking out loud.

for the nixpkgs repo (#749) i only get 4966 of (9150 + 32277 = 41427) issues = 12%
then git-bug bridge pull keeps saying imported 0 issues and 0 identities

i only get 4966 of (9150 + 32277 = 41427) issues = 12%

this looks very much like a limitation of the github API
which gives access only to the first 5000 items of a collection

also the github webinterface for NixOS/nixpkgs says

Issues 5k+
Pull requests 5k+

so we can use the github API to fetch the first 5000 items
and then we must fallback to webscraping to fetch the remaining items

possible workaround: https://github.com/settings/admin

Export account data
Export all repositories and profile metadata. Exports will be available for 7 days.

but this requires cooperation from the repo admins

Migration started.
Some repositories were too large to be included in the export.
Please visit the Migration API to export: nixpkgs

so the nixpkgs repo is too large for "Export account data"

fix: fetch-github-repo-data.sh
example result: milahu-nixpkgs-github-data

it seems clear to me that one way git-bug is used is for archival of external projects' issues. i think this is an important use case to address across the board. @milahu brings up a great point: we need to have a process for cleanly handling large issue counts from the bridges.

that said, i'm going to close this for now to help keep the active board clean. if you wish to work on this tree in the future, you can always check it out like so:

$ git fetch https://github.com/git-bug/git-bug.git refs/pull/760/head
$ git checkout FETCH_HEAD

it seems clear to me that one way git-bug is used is for archival of external projects' issues. i think this is an important use case to address across the board.

👍 on this. Are there any plans for such work to commense?

FWIW, my initial use for git-bug years ago was to prepare a "thorough" list + stats of contributors to datalad (and then heudiconv) for JOSS publication submission. Now also to facilitate access/review of issues by "corporate friends" (if you see @yarikoptic-gitmate -- must be one of those) without interrogating github API, and potentially as a bridge later for transfer or sync between github and forgejo(+aneksajo) instances. But some projects indeed have lots of "history" behind.

Would it be best to start off this PR rebased on master or ... ?