[RFC] selectively bridge bugs based on labels (and possibly other criteria) - #241

In term of UX, it make sense to me to only be able to configure that through the git config as it's a fairly narrow use case. A CLI command might be nice though.

Do you think a command to modify the configuration? Something like

git bug bridge config github.github-git-bug-test import-filter "..."

? I'm not sure that's any better than the existing:

git config git-bug.bridge.github.github-git-bug-test.import-filter "..."

I would probably just edit the config file either way.

Or are you thinking more of a command line option for bridge, like:

git bug bridge import --filter="..." --mod="..."

more like the last one, but we can figure that out later

i like the last example more, it gives the user the ability to filter issues without having to interact with the configuration.
Maybe we can use the configuration to store the default import filters/mods

About that, git-bug already has a way to express a filter: https://github.com/MichaelMure/git-bug/blob/master/doc/queries.md

This queries only applies to bug after they are imported though, which is not really what you are after. If you have millions of bugs in JIRA, you want to be able to request only the one you actually care about. BUT. What could be done here is to pass one of those query to the importers and have each implementation translate that into the bugtracker specific query/parameters/whatever that is needed to filter what is requested.

This would probably need some rework of how those query are parsed internally but it's doable. There is several benefits:

• those queries are already used in git-bug and will be familiar to the users
• those queries use the git-bug concepts and will apply to all bugtrackers, no bugtracker specific syntax
• those queries already cover quite a lot of concepts

Great! Yeah it totally makes sense to use the same filter mechanism as what is already there.

query to the importers and have each implementation translate that into the bugtracker specific query/parameters/whatever that is needed to filter what is requested.

100% agree. This was what my initial plan was. Then, after reading the code, I got lazy and figured it would be easier to just stack a filter on top of the existing code. At least to start. I think filtering *after the query, during the import, might still be required in general, for anything that a particular bugtracker doesn't support on through their API. But you're right about the JIRA situation. There would be way to many bugs to fetch over the web and then filter client-side.

This would probably need some rework of how those query are parsed internally

Why is that? Is it to expose an intermediate representation so that it's easier for the importer to translate it? Or to make it usable before the import? Either way, I'll poke around and see if I can grok the existing query/filter code.

At the moment, the query implementation is a single process translating something like status:open author:descartes sort:edit-asc into a Query, composed of Filter plus what is needed to order the result. Filter is basically a func(*BugExcerpt) bool, that is a function matching or not a bug.

To achieve what you want here, this process will need to be broken into two stages:

• some kind of micro parser, translating the string into an intermediary representation in memory
• a set of simple "compilers", translating that result into a Filter to act on BugExcerpt or generating the corresponding parameters/requests for each bridge.

Note that if a query can't be entirely translated into request parameters, a Filter as we have now can be applied to further filter before writing bugs in git.

Bridges in git-bug already tag the bugs with some metadata. Labels are typically a user-controlled data. Maybe what you really want is simply to surface in the UI some of those metadata ? Or tag bugs with more metadata ? Do you really need that to be user-controlled ?

Maybe what you really want is simply to surface in the UI some of those metadata ?

Possibly. I came up with the label-assignment scheme before I read the code so I hadn't realized that the git-bug already has some additional metadata that can be exploited.

Do you really need that to be user-controlled ?

For the export I think so. In my use case I would like to have multiple configured exporters I would like to be able to:

1. default export nothing
2. export anything "work" related to Jira,
3. export anything for "project X" to the github repository for X (for my use case, there are currently 11 X's)

So I think this I would like this to be user-controlled. Based on your feedback, however, one possible UI choice is: when I create a bug, in addition to assigning labels I can also assign a "remote" for that bug. This would be I think a minor UI addition and would require little extra change to the export logic and would work for any remote (agnostic of exporter). I think the UI action would just have to assign the additional metadata as-if the bug had been imported. We would have to make sure to assign the all the metadata required by the exporter to not get in a state the exporter is confused about, and probably modify the exporter slightly to recognize that "having some metadata but no ID" means that it was manually assigned. For this implementation I think I would like the exporter to not upload anything that lacks the manually added metadata. I guess in order to keep the current behavior, I can add a config option like export-untagged-bugs=False (the default =true being the current behavior).

I came up with the label filter idea because I am currently using labels to organize my bugs by project. Since I already have labels for each of the different "project X" and I figure that if we're already setting labels to categorize bugs than probably some combination of those labels will represent "exportable to X". So, I figured I would just add an extra label to indicate "exportable".

For the import I think the "auto assignment of labels" is important to me. As I mentioned above, I'm currently using labels to sort my bugs by project. So if I import issues from the "project X" github mirror, I would like them to automatically be tagged "project X" so that I can find them in the same set as bugs that I created for "project X".

Conclusion: I think that using the existing metadata as opposed to requiring "semantic use of labels" is a reasonable solution, and I like the idea of a UI change to "assign to remote".

Or tag bugs with more metadata ?

Possibly. In my exact use case listed above, I'm using labels to associate bugs to a project. A specific project piece of metadata would be nice and would enable the same features I'm looking for.

P.S. I was figuring that additional metadata support would be added over time as the project grew so I was thinking that generically filtering on or modifying arbitrary metadata would be a good way to future proof. I still think modifying data on import would be very useful. Perhaps modifying it again on export isn't necessary that metadata isn't being overloaded for exportability tagging.

P.P.S. Maybe a bug can be "assigned to" more than one "remote". That would be cool. Not something I am really needing right now but maybe someone could appreciate that. If nothing else, you could then use git-bug to bridge two bug systems. Wait, wasn't someone on gitter just asking about that? Hm...

^^^ Sorry for the text wall.

I think we generally agree on the requirement. The actual way to implement that is less clear though. There is some important UX consequences here so we should take the time to get that right.

A specific project piece of metadata would be nice and would enable the same features I'm looking for.

Isn't that essentially the same thing as the origin of the bug (bugtracker type + project) ?

I still think modifying data on import would be very useful. Perhaps modifying it again on export isn't necessary that metadata isn't being overloaded for exportability tagging.

My hope is that using the metadata system is enough, without additional user-controller tagging. This way, the labelling keep being its own thing, there is no need to cleanup anything before exporting, metadata being by nature a purely git-bug thing. They are stored with the bug though, which means they get pushed and pulled through a git remote. If there is no need for user-custom metadata this also stay its own thing.

Maybe a bug can be "assigned to" more than one "remote". That would be cool.

Yes, possibly, but I would wait for that to be obviously necessary.

If nothing else, you could then use git-bug to bridge two bug systems.

Maybe, but there is the limitation at the moment that a bridge can't export something with an identity (=user account) you don't have a token for. So you can't import everything from a bugtracker and push to another one.

Also a note: Query are not aware of metadata at the moment so it will be needed to expand that if that's the way to go.

Isn't that essentially the same thing as the origin of the bug (bugtracker type + project) ?

I wasn't originally thinking that "project" and "bugtracker type" would be a one-to-one mapping, but if you're advocating for that, I think it works fine for my needs. If ever there was a reason for "multi-remote" tagging that might be satisfied by "multi-project" assignment... so the idea seems to be scalable on the surface. It's maybe worth thinking about a couple of places where this model might not fit well and two come to mind:

1. monorepo (code) composed of multiple projects
2. mono-bugtracker composed of multiple projects - i.e. JIRA for a medium to large sized company

I don't think either of these usecases prevents the project:bucktracker 1-1 mapping though. It might just mean that you need to configure two (or more) bridges that point to the same remote bugtracker (one per "project") which I am fine with.

In any case, it certainly works for me. Although...

My hope is that using the metadata system is enough, without additional user-controller tagging.

My preference would be that anything I create in git-bug would not "export" automatically, unless I ask it to. I would, however, still like to associate it with a particular project so that I can see all bugs relevant to that project in the termui and such, so I think I would still like it to have a "project assignment".

To be a little more concrete, let's talk specifically about "Project X" and let's assume that we have project as a new field in git-bug (or at least exposed through the UI, possibly implemented using existing metadata). Futhermore, let's assume that I have a "Project X" mirror on github with "issues" enabled so that users can report issues, but that git-bug is the "canonical" bug tracker. If a user of Project X reports a bug on github, I would like that bug to be imported into git-bug and assigned the project=X metadata. That way I can find it with all my other project=X bugs. However, I don't necessarily want all the git-bugs with the project=X label to be exported back to github.

I can respect that maybe others would want them to automatically go to github, but I would prefer to explicitly choose.

That said, I'm not fixated on using labels for this purpose. If project were available as a piece of metadata, maybe what I'm asking for is an addition boolean flag called public=true, or maybe private=false or maybe export=true (by default, whereas for my preference I would config it to be the opposite). I guess the semantics would be: Any bug whose origin is the remote tracker is private=false, any bug whose origin is git-bug is private=<user-config,default=false> initially. Then add some UI to flip the flag.

It make sense to me to be able to control if you want to export "free floating" bugs. However as the majority of users will have one repo for one external bugtracker, it make sense to keep exporting everything as the default.

It might just mean that you need to configure two (or more) bridges that point to the same remote bugtracker (one per "project") which I am fine with.

At the moment, bridges will tag bugs with the target (github, gitlab ..) and another specific information (the project URL for github). So having multiple bridges for the same bugtracker won't help you.

However, in the case of monorepo or mono-bugtracker, I expect either:

• remote bugs have a project specific label (that is, a user specific way to classify them), so you can filter based on that
• bugs are in some sort of global pool with a bugtracker specific way to classify them (a project field or something), so the bridge implementation could use that to define what is a project. This way, you could have multiple bridges with the same remote bugtracker, one for each project you care about. Bugs would be tagged distincltively.

However, in the case of monorepo or mono-bugtracker, I expect either: ...

Think this is also what I was thinking.

Do you think a command to modify the configuration? Something like

git bug bridge config github.github-git-bug-test import-filter "..."

? I'm not sure that's any better than the existing:

git config git-bug.bridge.github.github-git-bug-test.import-filter "..."

I would probably just edit the config file either way.

Or are you thinking more of a command line option for bridge, like:

git bug bridge import --filter="..." --mod="..."

Great! Yeah it totally makes sense to use the same filter mechanism as what is already there.

query to the importers and have each implementation translate that into the bugtracker specific query/parameters/whatever that is needed to filter what is requested.

100% agree. This was what my initial plan was. Then, after reading the code, I got lazy and figured it would be easier to just stack a filter on top of the existing code. At least to start. I think filtering *after the query, during the import, might still be required in general, for anything that a particular bugtracker doesn't support on through their API. But you're right about the JIRA situation. There would be way to many bugs to fetch over the web and then filter client-side.

This would probably need some rework of how those query are parsed internally

Why is that? Is it to expose an intermediate representation so that it's easier for the importer to translate it? Or to make it usable before the import? Either way, I'll poke around and see if I can grok the existing query/filter code.

Maybe what you really want is simply to surface in the UI some of those metadata ?

Possibly. I came up with the label-assignment scheme before I read the code so I hadn't realized that the git-bug already has some additional metadata that can be exploited.

Do you really need that to be user-controlled ?

For the export I think so. In my use case I would like to have multiple configured exporters I would like to be able to:

1. default export nothing
2. export anything "work" related to Jira,
3. export anything for "project X" to the github repository for X (for my use case, there are currently 11 X's)

So I think this I would like this to be user-controlled. Based on your feedback, however, one possible UI choice is: when I create a bug, in addition to assigning labels I can also assign a "remote" for that bug. This would be I think a minor UI addition and would require little extra change to the export logic and would work for any remote (agnostic of exporter). I think the UI action would just have to assign the additional metadata as-if the bug had been imported. We would have to make sure to assign the all the metadata required by the exporter to not get in a state the exporter is confused about, and probably modify the exporter slightly to recognize that "having some metadata but no ID" means that it was manually assigned. For this implementation I think I would like the exporter to not upload anything that lacks the manually added metadata. I guess in order to keep the current behavior, I can add a config option like export-untagged-bugs=False (the default =true being the current behavior).

I came up with the label filter idea because I am currently using labels to organize my bugs by project. Since I already have labels for each of the different "project X" and I figure that if we're already setting labels to categorize bugs than probably some combination of those labels will represent "exportable to X". So, I figured I would just add an extra label to indicate "exportable".

For the import I think the "auto assignment of labels" is important to me. As I mentioned above, I'm currently using labels to sort my bugs by project. So if I import issues from the "project X" github mirror, I would like them to automatically be tagged "project X" so that I can find them in the same set as bugs that I created for "project X".

Conclusion: I think that using the existing metadata as opposed to requiring "semantic use of labels" is a reasonable solution, and I like the idea of a UI change to "assign to remote".

Or tag bugs with more metadata ?

Possibly. In my exact use case listed above, I'm using labels to associate bugs to a project. A specific project piece of metadata would be nice and would enable the same features I'm looking for.

P.S. I was figuring that additional metadata support would be added over time as the project grew so I was thinking that generically filtering on or modifying arbitrary metadata would be a good way to future proof. I still think modifying data on import would be very useful. Perhaps modifying it again on export isn't necessary that metadata isn't being overloaded for exportability tagging.

P.P.S. Maybe a bug can be "assigned to" more than one "remote". That would be cool. Not something I am really needing right now but maybe someone could appreciate that. If nothing else, you could then use git-bug to bridge two bug systems. Wait, wasn't someone on gitter just asking about that? Hm...

^^^ Sorry for the text wall.

just a note, it will likely be easier to just translate all the query into the already existing filter and apply that again no matter what. It's super fast anyway.

those are tricky trade-off ...I need to think more about that. One thing that is clear though is that "project" is a UX concept and doesn't necessarily have to translate into a "project" tag on a bug. What would be possible is to say that a "project" is a set of metadata (target + project ID for gitlab, target + ... for github ..). As you can see, this would be bridge specific but that can work. Each configured bridge would advertise to the rest of git-bug their "project" so the termui for instance could tag before import/export.

Another question, does a project is a property of a bug (meaning shared by everyone push/pulling) or is it a personal choice ? If it's the later, it can't be done using metadata as they are transferred with the bugs.

Not sure as well if "project" is the best term. it works for you but is it universal enough ?

If we consider that "project" is a personal choice, we need a local way to map bugs to sources. A downside of this is that if this mapping is lost it can't really be reconstructed from the bug database. Also, another user getting the bugs through git will not have this mapping either.

^ this is really a key question for this design

be the change you want to see in this world :-D

Tricky question again. Some of those will be first-class citizen (= have their own Operation) in git-bug (like the quite obvious assignee). But at the end of the day git-bug can't support every possible workflow or every field existing in other bugtrackers, so choices (or plugin, or fork) will need to be made.

git-bug is designed in a way that bugs are still readable even if an operation type is unknown so it's fine if not everything client support everything, those operations will just be ineffective.

There is also the idea of having a new kind of Operation to store arbitrary fields. @seeduvax is working on that at https://github.com/seeduvax/git-bug/tree/user-defined-attribute

All of that to say that metadata are probably not the right vehicle for those informations. Metadata are more destined to support internal data without history.