Implement jira bridge by cheshirekow · Pull Request #250 · git-bug/git-bug

cheshirekow · 2019-11-14T18:04:00Z

No description provided.

cheshirekow · 2019-11-14T18:05:29Z

Some notes:

I suggest you start with doc/jira_bridge.md as it describes several of the jira-specific issues that are dealt with by the bridge, as well as some (initial) design decisions.

Note that I've tested the import quite a bit, but I've only tested the export with one bug that I created, exported, imported, modified, exported again. I don't want to use my work JIRA instance in case I messed up and export thousands duplicate bugs into our jira (the project I used to test has 6000 issues). One thing that would help a lot is if we could get in touch with Atlassian and request that they donate a license (or a cloud instance) to the git-bug project for testing . I think we really need a mock server for CI,but for dev cycles it would definitely be nice to work with the real thing. AlsoI think we should really test the export a bit before putting this out in the world. I would be happy to try and contact Atlassian, but I think as the project owner they might appreciate more hearing from @MichaelMure.

The bridge currently does nothing to import any of the JIRA fields that don'thave git-bug equivalents ("Assignee", "sprint", "story points", etc). I'm not sure what the right way to go about this is, but it would really be awesomeif git-bug could support all of these fields. I would love to hear thoughts on this for future development.

I don't fully understand what the whole getIdentityClient code is for in the existing plugins. I have a suspicion it's there for future use (the ability to assume different identities and act on behalf of those identities). I tried to keep that stuff around in the JIRA bridge but that code isn't really doinganything right now.

MichaelMure · 2019-11-14T18:34:30Z

I think we really need a mock server for CI,but for dev cycles it would definitely be nice to work with the real thing. AlsoI think we should really test the export a bit before putting this out in the world. I would be happy to try and contact Atlassian, but I think as the project owner they might appreciate more hearing from @MichaelMure.

Why not using a free plan, hosted by Atlassian ? It's up to 10 seat, "free forever".

The bridge currently does nothing to import any of the JIRA fields that don'thave git-bug equivalents ("Assignee", "sprint", "story points", etc). I'm not sure what the right way to go about this is, but it would really be awesomeif git-bug could support all of these fields. I would love to hear thoughts on this for future development.

I thouched a bit on that here: #241 (comment)

I don't fully understand what the whole getIdentityClient code is for in the existing plugins. I have a suspicion it's there for future use (the ability to assume different identities and act on behalf of those identities). I tried to keep that stuff around in the JIRA bridge but that code isn't really doing anything right now.

Yeah, it doesn't really do anything at the moment but will eventually. The goal is to be able to load as much credential as necessary and being able to use git-bug as a "real" bridge between two bug tracker for a migration. It's not really part of git-bug's end goal but it was a fairly simply architectural change to enable that use case.

MichaelMure · 2019-11-14T18:52:16Z

Huu, sorry for asking that after you wrote that much code, but have you looked at https://github.com/andygrunwald/go-jira ? Wouldn't that simplify a bunch of things in here ?

cheshirekow · 2019-11-14T21:46:38Z

Why not using a free plan, hosted by Atlassian ? It's up to 10 seat, "free forever".

For real? Wow awesome. I only found the 30 day server trial. I'll look into this.

I thouched a bit on that here

Ok great, I haven't gotten back to that discussion yet. I've been focused on this bridge :).

Yeah, it doesn't really do anything at the moment but will eventually.

Ok good. I think I've respected the setup and the plumbing is in place for this bridge as well.

but have you looked at https://github.com/andygrunwald/go-jira

Nope. It's news to me.

Wouldn't that simplify a bunch of things in here ?

I don't think so... I mean it would be a lot less code to use someone else's REST wrapper, if that's what you mean, but I don't think the rest of the bridge implementation changes. If that library looks good I would be happy to swap out for it.

sorry for asking that after you wrote that much code

..meh, the big time sink was studying the API and testing behaviors. Writing the REST wrapper was relatively quick.

MichaelMure · 2019-11-15T00:01:37Z

I mean it would be a lot less code to use someone else's REST wrapper

Having someone else do the maintenance is a big win, at least for me ;)

MichaelMure

That's a lot of code ! Nothing obviously wrong, great start !

MichaelMure · 2019-11-15T14:34:51Z

For real? Wow awesome. I only found the 30 day server trial. I'll look into this.

I created an instance at https://git-bug.atlassian.net give me an email so I can invite you.

cheshirekow

I think the github review interface is buggy. I thought it discarded some of my replies, but then I think they may have come back. I apologize if there are duplicates. I put a 👍 on some of your requested changes to acknowledge them. I will comment with "Done" when they are done.

cheshirekow · 2019-11-18T20:57:20Z

+entry to our git-config:
+
+```
+create-issue-defaults = {"issuetype":"10001"}


For the raw JSON request? No it must be an ID.
For git-bug? Sure it could but there are a couple of considerations:

The mapping can change more easily than the workflow. If a project manager decides that "Open" should be renamed to "Initiated" the textual name will change but the ID and workflow specification will stay the same.

We'll need to lookup the mapping and translate it when we merge the JSON into our outgoing request

Issue types are not namespaced by project, they are global to a JIRA server so all projects share a common pool of names and ids. This may or may not be an issue if the pool is large. I don't know how big the mapping might get. The large company I currently work for has 155 issue types that are visible to me, which is small enough to query once and reuse.

If you feel like it's worthwhile we can go down that path. I would advocate to make this a future enhancement.

cheshirekow · 2019-11-18T20:59:16Z

+You can specify the mapping between `git-bug` status and JIRA status id's using
+the following:
+```
+bug-open-id = 1


That's a good point. This should really be a dictionary style mapping.

cheshirekow · 2019-11-18T21:15:09Z

For real? Wow awesome. I only found the 30 day server trial. I'll look into this.

I created an instance at https://git-bug.atlassian.net give me an email so I can invite you.

Sent you an email. I'm glad you were able to get one. I've been trying for a few days now and I keep getting an error. Looks like it worked for me today though.

cheshirekow

Alright I think I've addressed all the comments so far. I'm out of time for the evening so I will test the changes and push an update to this PR when I get a chance.

cheshirekow · 2019-11-23T06:43:17Z

+	conf[core.KeyTarget] = target
+	conf[keyServer] = serverURL
+	conf[keyProject] = project
+	if choice == 1 {


cheshirekow · 2019-11-23T06:53:36Z

+}
+
+// PromptPassword performs interactive input collection to get the user password
+func PromptPassword() (string, error) {


Moved to input/prompt.go. We can follow up with using it in other bridges (TODO 3cb989e)

cheshirekow · 2019-11-23T07:38:55Z

+	targetOpID, err := b.ResolveOperationWithMetadata(
+		keyJiraID, item.ID)
+	if err == nil {
+		self.out <- core.NewImportNothing("", "comment already imported")


Ok, done. Removed the superfluous nothing events. One question though: if we can't import/export a particular operation due to lack of credentials or for some other criteria, should we silently move on or should we emit an error event? Same question as in the exporter above really.

cheshirekow · 2019-11-23T07:43:12Z

 }

+// Convenience function to edit the body of a bug (the first comment)
+func EditBody(b Interface, author identity.Interface, unixTime int64, message string) (*EditCommentOperation, error) {


Done. Also made the same edits in bug_cache.go

cheshirekow · 2019-11-23T07:49:26Z

+
+## Configuration
+
+As mentioned in the notes above, there are a few optional configuration fields


cheshirekow · 2019-11-23T07:52:09Z

+entry to our git-config:
+
+```
+create-issue-defaults = {"issuetype":"10001"}


Cool. Added TODO b610296 to look into this.

cheshirekow · 2019-11-23T08:04:16Z

+You can specify the mapping between `git-bug` status and JIRA status id's using
+the following:
+```
+bug-open-id = 1


MichaelMure · 2019-12-04T16:00:21Z

So much teasing and no push ;)

cheshirekow · 2019-12-05T00:56:26Z

So much teasing and no push ;)

Ah, yeah sorry. I can push when I am home tonight. I wanted to test and then report on the test results with the next update, but now I have to deal with this:

On one hand, it's good they support user API tokens... On the other hand, we'll probably need both code paths because companies are slow to upgrade their JIRA deployments.

cheshirekow · 2019-12-05T06:58:20Z

I did a little testing with my jira cloud instance and things are looking good. I did:

create a git-bug with a label, export to jira
add a comment in jira, import to git-bug
edit the description in git-bug, push to jira

I had to hack in support for the new jira token auth. I'll need to review the new git-bug token management API to get that plumbed in correctly (not hacky) though... and do a little more comprehensive testing. Feel free to comment further on the current state of the patch though. I don't think too much will change to get in the token auth support.

MichaelMure · 2019-12-05T11:53:03Z

but now I have to deal with this

Ha bummer :( Github is doing that as well ... https://developer.github.com/changes/2019-11-05-deprecated-passwords-and-authorizations-api/

MichaelMure

Just a few things I noticed. Generally look very good, great work :)

As you mentioned, credentials stored through the core and a bit of standardization with the other bridges would be nice.

Automated testing against the jira instance, as it is done with github and gitlab would be very useful as well.

MichaelMure · 2019-12-08T13:35:16Z

I'm working on generalizing the credential code in the core so it's easy to add more kind.

cheshirekow

Ok, unhackified the two credentials-types jazz. I saw that you created a PR for a new credentials system for the core so I will take a look at that. I believe that all the review comments are addressed at this point. Pushing now...

Edit: here is my list of follow-up work:

8391e86 create unit tests against test instance
1121826 Add unit tests for utilities
0cf5c71 Assign git-bug to jira field on import
c70e22a implement additional query filters for import
8acce9c Download and cache workflow representation
67bf520 After JIRA mock, create integration tests
9ecefaa Create JIRA mock and add REST unit tests
0597088 use OS keyring for credentials
d3e8f79 You shouldn't count on the Total value in paginations

cheshirekow · 2019-12-09T17:40:23Z

+)
+
+const (
+	keyOrigin          = "origin"


cheshirekow · 2019-12-09T17:46:22Z

+// Create a bug.Comment from a JIRA comment
+func (self *jiraImporter) ensureComment(
+	repo *cache.RepoCache, b *cache.BugCache, item Comment) error {
+	// ensure person


Oh interesting. In the languages I usually work with splitting arguments at different indention would be frowned upon. Good to know though, I'll go ahead and fix this one here and try to follow that moving forward.

golangcibot · 2019-12-09T18:04:19Z

+	password := conf[keyPassword]
+	if password == "" {
+		var err error
+		password, err = input.PromptPassword()


ineffectual assignment to password (from ineffassign)

cheshirekow · 2019-12-11T18:16:34Z

Did some testing last night and fixed a couple of bugs. I still haven't rebased on the new credential system, so I guess that is up next. Things are looking good so far though and I think will start using this at work to shake out more bugs and inspire a test plan.

I'm going to resolve the open conversations on the review and make sure I didn't miss anything.

cheshirekow · 2019-12-13T21:38:41Z

Update #5: I had to fix the git config parser in the core in order to correctly parse out a JSON map as a git config value. I also added the import/export warning events and used them. Lastly, I added a "reverse map" for bug ids, so that git-bug knows how to interpret JIRA statuses on import (if there are many).

MichaelMure · 2020-02-05T13:15:38Z

Hey @cheshirekow what's up with this ? Do you need help ?

cheshirekow · 2020-02-05T15:33:17Z

I rebased last week and resolved conflicts but it doesn't build due to interface requirements on the bridge. I guess you changed the bridge API? I haven't had time to dig in further.

As of the last commit all your comments have been addressed I think. It's fully functional and I've been using this branch as my daily driver for some weeks now.

Some help adapting to whatever the upstream changes were would be great!

MichaelMure · 2020-02-05T15:41:13Z

Alright, I'll finish #294 and rebase/merge this.

If you have the time, an automated test similar as the other bridges would be great.

MichaelMure · 2020-02-09T21:24:06Z

First merge, use of the new prompts and some cleanup. This need more work especially on the credential part.

cheshirekow · 2020-02-12T16:30:13Z

Thanks for the push. I built this morning and my pull/push from JIRA worked just fine.

MichaelMure · 2020-02-15T13:22:40Z

@cheshirekow do you think it would be possible to detect automatically the credential (session or token) ? That would simplify both the code and the UX.

MichaelMure · 2020-02-15T13:50:02Z

Also, there is a mismatch between the name and value of this constant, what's the good concept ? metaKeyJiraOperationId = "jira-derived-id"

cheshirekow · 2020-02-17T21:36:28Z

@cheshirekow do you think it would be possible to detect automatically the credential (session or token) ? That would simplify both the code and the UX.

Your idea in the other thread of trying some query with both credential types and just seeing what works makes sense.

Also, there is a mismatch between the name and value of this constant, what's the good concept ? metaKeyJiraOperationId = "jira-derived-id"

As I mentioned in the other thread, I'm not sure which is better to use, but I'm happy with either. Which do you prefer? I might suggest changing the name of the variable and keeping the metadata key just so I don't have to re-import my 1500-ish bugs that are already using "jira-derived-id" in their metadata. I'll go ahead and make that change.

cheshirekow · 2020-02-17T21:44:48Z

Successful import from git-bug.atlassian.net and from my work jira so things look good with your credential changes. Thanks for pulling those in.

MichaelMure · 2020-02-18T18:34:19Z

But what if you need to use two different account for two different bridges?

So, that would only matter for importing a private repo. For importing a public repo, any credential would do. For exporting, the bridge will match credentials based on the login, using whatever credential is needed for this specific piece of data.

So what could be done is for the bridges to store an additional information in their config: the login to use when importing (ie, the one used during the configuration). That would make more sense than using the first random one found.

MichaelMure · 2020-02-19T11:14:28Z

So, does that means it's merge time ? What's left to do here ?

MichaelMure · 2020-02-19T15:06:46Z

Ho, at least one thing would be to complete the Readme

cheshirekow · 2020-02-19T17:40:01Z

So, does that means it's merge time ? What's left to do here ?

Ready from my end.

Ho, at least one thing would be to complete the Readme

You mean add "Jira" to the tables? Ok, I just pushed a change to do that.

MichaelMure · 2020-02-19T18:07:58Z

Weeeeeeeeeee 🎉 🎉 🎉 🎉 🎉 🎉

golangcibot reviewed Nov 14, 2019

View reviewed changes

MichaelMure requested changes Nov 15, 2019

View reviewed changes

cheshirekow commented Nov 18, 2019

View reviewed changes

cheshirekow mentioned this pull request Nov 23, 2019

Bridge: correctly emit NothingEvents #254

Merged

cheshirekow commented Nov 23, 2019

View reviewed changes

cheshirekow force-pushed the cheshirekow-jira branch from 056e90c to 0ef9b80 Compare December 5, 2019 06:47

MichaelMure reviewed Dec 5, 2019

View reviewed changes

Comment thread bridge/jira/export.go

Comment thread bridge/jira/import.go Outdated

Comment thread bridge/jira/export.go Outdated

MichaelMure mentioned this pull request Dec 6, 2019

question: is anyone working on a JIRA bridge? #270

Closed

cheshirekow commented Dec 9, 2019

View reviewed changes

cheshirekow requested a review from MichaelMure December 9, 2019 18:00

golangcibot reviewed Dec 9, 2019

View reviewed changes

cheshirekow force-pushed the cheshirekow-jira branch from d4c6fff to 262102b Compare December 9, 2019 18:30

MichaelMure added this to the 0.7.0 milestone Dec 11, 2019

cheshirekow force-pushed the cheshirekow-jira branch from 8862a2e to 1e23ffd Compare December 13, 2019 21:55

cheshirekow added 2 commits December 18, 2019 07:42

Implement jira bridge

eff830b

codereview git-bug#1: clean lint from golangcibot

7657a38

MichaelMure added 3 commits February 9, 2020 20:23

Merge remote-tracking branch 'origin/master' into cheshirekow-jira

b3d3612

jira: use the new generalized prompts

2792c85

jira: admittedly biased go styling

b2ca506

MichaelMure added 3 commits February 14, 2020 22:56

Merge remote-tracking branch 'origin/master' into cheshirekow-jira

e9aff2a

Merge remote-tracking branch 'origin/master' into cheshirekow-jira

432a816

Merge remote-tracking branch 'origin/master' into cheshirekow-jira

cfa0058

MichaelMure and others added 5 commits February 15, 2020 15:51

Merge remote-tracking branch 'origin/master' into cheshirekow-jira

0bb9ed9

jira: rework to use the credential system + adapt to refactors

5c230cb

jira: fix a nil context

01b0a93

Fix jira bridge config didn't save credentials or store URL metadata

bbd31ad

fix usage of newIdentityRaw

d349137

metaKeyJiraOperationId -> metaKeyJiraDerivedId

4827d3f

update README

be763b4

MichaelMure merged commit b331833 into git-bug:master Feb 19, 2020

This was referenced Feb 21, 2020

bridges should remember the login to use for importing #338

Closed

bridge: record the login used during the configure and use it as default credential #341

Merged


		## Configuration

		As mentioned in the notes above, there are a few optional configuration fields

Conversation

cheshirekow commented Nov 14, 2019

Uh oh!

cheshirekow commented Nov 14, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MichaelMure commented Nov 14, 2019

Uh oh!

MichaelMure commented Nov 14, 2019

Uh oh!

cheshirekow commented Nov 14, 2019

Uh oh!

MichaelMure commented Nov 15, 2019

Uh oh!

MichaelMure left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MichaelMure commented Nov 15, 2019

Uh oh!

cheshirekow left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cheshirekow commented Nov 18, 2019

Uh oh!

cheshirekow left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MichaelMure commented Dec 4, 2019

Uh oh!

cheshirekow commented Dec 5, 2019

Uh oh!

cheshirekow commented Dec 5, 2019

cheshirekow left a comment •

edited

Loading

MichaelMure commented Feb 18, 2020 •

edited

Loading