Repository navigation
Rate limit reached when pulling from a bridge #749
Description
Ten minutes ago, trying to git bug bridge pull the nixpkgs project's bugs (>5k bugs, >3k PRs -- a very very large set of bugs):
import error: Something went wrong while executing your query. Please include `A8DE:54BE:87BAE:9F6E7:62140406` when reporting this issue.
I have never been able to get a pull of nixpkgs' bugs to complete, and I've been trying for weeks now (see also #740).
I am using 05d73e1, which has the fix for #585.
Is there any chance that git-bug could keep fetching other bugs when it encounters a problem like this? It appears that it just aborts as soon as any API call produces an error. The nixpkgs bugset is ginormous, so it's going to take me at least a week (due to ratelimits) anyways. But right now it keeps failing after 108 bugs out of 5000+.
Is there an environment variable I can set to cause git-bug to print a backtrace or other context when it gives up like this?
Activity
rng-dynamics commentedon Feb 27, 2022
The error is from the github server. But I can tell you what I would try:
You should put the nixpkg repo in a tmpfs (
mount -t tmpfs) because there is a big number of issues in nixpkg and git-bug writes more data than necessary on your disk while importing. Using a tmpfs might help you to speed up the import.You can try to decrease the size of the queries to the github server here: https://github.com/MichaelMure/git-bug/blob/b2f0e126a10b5f030bfc35cac4b0cabcf083b589/bridge/github/import_mediator.go#L12-L15
These are the number of items requested from the server with one query. You could start by reducing
NumIssues. If it doesn't work, try to reduce the other ones, too. This will slow the import down, but it might enable you to get a full import without error. If you have success with this method, please let me know which numbers you used.You can increase the number of retries in case the github server has an error:
https://github.com/MichaelMure/git-bug/blob/b2f0e126a10b5f030bfc35cac4b0cabcf083b589/bridge/github/client.go#L110
I hope that helps. Let us know how it is going.
MichaelMure commentedon Feb 27, 2022
We should adress that at some point, any idea why that is?
rng-dynamics commentedon Feb 28, 2022
Do you remember that discussion: #585 (comment). As far as I understand
.git/git-bug/bug-cacheand.git/git-bug/identity-cacheare rewritten after every imported issue, comment, edit, ... .MichaelMure commentedon Feb 28, 2022
Ha yes, I forgot about that, thanks.
rng-dynamics commentedon Mar 1, 2022
I will have look at it, but I will need a few days. In case I do not come back to this issue in about a week, please feel free to ping me.
MichaelMure commentedon Mar 1, 2022
@rng-dynamics it might make sense to implement another
ImportResult/ExportResultfor simple logs that the bridges could use to spit out more detailed information on what's happening. Those could be made visible with a--verboseflag or something like that.At this point it would make sense to rename those to
importEvent/ExportEvent.rng-dynamics commentedon Mar 9, 2022
@a-m-joseph, thanks for the bug report. The errors are seemingly arbitrary errors by github. If you want you can use my draft in #760. Even if there are github errors with #760 the bridge will continue importing the other issues. And you can run the import again and it will try to fetch the remaining issues. Have look at the file
git-bug-importafter running the import. The content of the file should be self-explanatory.11 remaining items
github-actions commentedon Jul 23, 2024
This bot triages untriaged issues and PRs according to the following rules:
lifecycle/stalelabel is appliedlifecycle/stalewas applied, the issue is closedTo remove the stale status, you can:
lifecycle/stalelabelgithub-actions commentedon Jan 22, 2025
This bot triages issues in order to help the maintainers identify what
needs attention, according to the following lifecycle rules:
lifecycle/staleis appliedlifecycle/stalewas applied,lifecycle/rottenis appliedThis bot will not automatically close stale issues.
To remove the stale status, you can:
To avoid automatic lifecycle management of this issue, add
lifecycle/frozen.sudoforge commentedon Apr 28, 2025
As a stop-gap to solve this and any related issues (i'll link any i find in this comment later; i remember seeing this pop up a few times), an "infinite" incremental backoff can be implemented for all operations that pull from a bridge. Whether or not this is actually infinite, or if it kills the process after a certain number of retries is TBD. I think it would be reasonable to exit if we get a known response back that equates to "you have reached the rate limit".
[-]problems importing from github.com/nixos/nixpkgs[/-][+]Rate limit reached when pulling from a bridge[/+]