Repository navigation
Github bridge import: error, probably caused by line of 65536+ length #606
Description
I tried to import all the issues from https://github.com/golang/go. (Currently I am using git-bug from PR #585).
This gives the following error:
import error: issue creation: key too large
imported 18123 issues and 5456 identities with default bridge
The error is caused when importing golang/go#18201 .
This issue contains a line of 65536+ length.
Activity
MichaelMure commentedon Mar 27, 2021
Hu, that's a funny one.
Are you able to pinpoint a bit more where this error happen?
rng-dynamics commentedon Mar 27, 2021
At the moment I do not know where the error happens exactly. Just looking at the error message it should originiate from somewhere below the call to
repo.NewBugRaw().The cause might be the same as described in the issue which causes the error. There the cause was a
bufio.Scanner(which hasbufio.MaxScanTokenSizeequal to 65536). The solution which the golang people found was to replace thebufio.Scannerby abufio.Readeras described in https://go-review.googlesource.com/c/tools/+/93439/MichaelMure commentedon Mar 28, 2021
So, what's happening is that this text is passed to Bleve (full text search) for indexing, Bleve process it then try to write a piece of data (with a very long key) to BoltDB (a key/value store).
I filled an issue here: blevesearch/bleve#1576
MichaelMure commentedon Mar 28, 2021
I suppose that as a workaround, long token could be filtered before indexing in here: https://github.com/MichaelMure/git-bug/blob/fb6b2d873e1bbc653aa84ddce56c842202e00c33/cache/repo_cache_bug.go#L482-L484
MichaelMure commentedon Mar 28, 2021
Should be fixed with 32958b5