Skip to content

allow allow contribute a search filter in jira bridge #505

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
nxmatic wants to merge 6 commits into from
Closed

allow allow contribute a search filter in jira bridge #505

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
69dd0f2
allow contribue search filter in jira bridge
Nov 28, 2020
5da77b1
fix jira config override without anti-slash (let store jira params ou…
Nov 28, 2020
f3da9fb
trim user fields before validation
Nov 28, 2020
d2bcafb
fix NPE when there is no bridge configuration in global settings
Nov 28, 2020
9213a4e
read last import timestamp from bridge configuration (already loaded …
Nov 28, 2020
9dfef20
filter bugs using the bridge name for export in jira bridge
Nov 29, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions bridge/core/bridge.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -263,7 +263,7 @@ func (b *Bridge) ensureConfig() error {
func loadConfig(repo repository.RepoConfig, name string) (Configuration, error) {
keyPrefix := fmt.Sprintf("git-bug.bridge.%s.", name)

pairs, err := repo.LocalConfig().ReadAll(keyPrefix)
pairs, err := repo.AnyConfig().ReadAll(keyPrefix)

@MichaelMure MichaelMure Nov 28, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why would you need to read the global config? The bridge config only make sense and is stored locally.

@nxmatic nxmatic Nov 29, 2020

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For sharing bridges definition in multiple projects. This way I can define them only once for the multiple repositories we're working on in the same jira project.

if err != nil {
return nil, errors.Wrap(err, "error while reading bridge configuration")
}
Expand Down Expand Up @@ -300,7 +300,7 @@ func (b *Bridge) ensureImportInit(ctx context.Context) error {

importer := b.getImporter()
if importer != nil {
err := importer.Init(ctx, b.repo, b.conf)
err := importer.Init(ctx, b.repo, b.Name, b.conf)
if err != nil {
return err
}
Expand All @@ -317,7 +317,7 @@ func (b *Bridge) ensureExportInit(ctx context.Context) error {

exporter := b.getExporter()
if exporter != nil {
err := exporter.Init(ctx, b.repo, b.conf)
err := exporter.Init(ctx, b.repo, b.Name, b.conf)
if err != nil {
return err
}
Expand Down Expand Up @@ -376,12 +376,19 @@ func (b *Bridge) ImportAllSince(ctx context.Context, since time.Time) (<-chan Im

func (b *Bridge) ImportAll(ctx context.Context) (<-chan ImportResult, error) {
// If possible, restart from the last import time
lastImport, err := b.repo.LocalConfig().ReadTimestamp(fmt.Sprintf("git-bug.bridge.%s.lastImportTime", b.Name))
if err == nil {
return b.ImportAllSince(ctx, lastImport)
since, err := b.lastImportTime()
if err != nil {
return nil, err
}
return b.ImportAllSince(ctx, since)
}

return b.ImportAllSince(ctx, time.Time{})
func (b *Bridge) lastImportTime() (time.Time, error) {
lastImport := b.conf["lastImportTime"]
if lastImport == "" {
return time.Time{}, nil
}
return repository.ParseTimestamp(lastImport)
}

func (b *Bridge) ExportAll(ctx context.Context, since time.Time) (<-chan ExportResult, error) {
Expand Down
4 changes: 2 additions & 2 deletions bridge/core/interfaces.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,11 @@ type BridgeImpl interface {
}

type Importer interface {
Init(ctx context.Context, repo *cache.RepoCache, conf Configuration) error
Init(ctx context.Context, repo *cache.RepoCache, name string, conf Configuration) error

@MichaelMure MichaelMure Nov 28, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why you would need to pass the bridge name. This name is purely arbitrary for the user to identify configured bridges, so it's not valuable for anything else. Am I missing something?

@nxmatic nxmatic Nov 29, 2020

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For replacing the project id, will be used for the export filtering in the bridge. I missed that change, will update the pull request with.

@nxmatic nxmatic Nov 29, 2020

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about sharing the bridges configuration in the repository itself. All the contributors are using the same no ? This way when a contributor clone the repository, he'll get the associated bridges with. He will only have to give his credentials.

ImportAll(ctx context.Context, repo *cache.RepoCache, since time.Time) (<-chan ImportResult, error)
}

type Exporter interface {
Init(ctx context.Context, repo *cache.RepoCache, conf Configuration) error
Init(ctx context.Context, repo *cache.RepoCache, name string, conf Configuration) error
ExportAll(ctx context.Context, repo *cache.RepoCache, since time.Time) (<-chan ExportResult, error)
}
3 changes: 3 additions & 0 deletions bridge/core/params.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ type BridgeParams struct {
TokenRaw string // pre-existing token to use (Github, Gitlab, , )
Owner string // owner of the repo (Github, , , )
Project string // name of the repo or project key (Github, , Jira, Launchpad)
Filter string // import filter ( , . Jira, )
}

func (BridgeParams) fieldWarning(field string, target string) string {
Expand All @@ -30,6 +31,8 @@ func (BridgeParams) fieldWarning(field string, target string) string {
return fmt.Sprintf("warning: --owner is ineffective for a %s bridge", target)
case "Project":
return fmt.Sprintf("warning: --project is ineffective for a %s bridge", target)
case "Filter":
return fmt.Sprintf("warning: --filter is ineffective for a %s bridge", target)
default:
panic("unknown field")
}
Expand Down
2 changes: 1 addition & 1 deletion bridge/github/export.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ type githubExporter struct {
}

// Init .
func (ge *githubExporter) Init(_ context.Context, repo *cache.RepoCache, conf core.Configuration) error {
func (ge *githubExporter) Init(_ context.Context, repo *cache.RepoCache, name string, conf core.Configuration) error {
ge.conf = conf
ge.identityClient = make(map[entity.Id]*githubv4.Client)
ge.cachedOperationIDs = make(map[entity.Id]string)
Expand Down
4 changes: 2 additions & 2 deletions bridge/github/export_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ func TestPushPull(t *testing.T) {

// initialize exporter
exporter := &githubExporter{}
err = exporter.Init(ctx, backend, core.Configuration{
err = exporter.Init(ctx, backend, "test", core.Configuration{
confKeyOwner: envUser,
confKeyProject: projectName,
confKeyDefaultLogin: login,
Expand Down Expand Up @@ -217,7 +217,7 @@ func TestPushPull(t *testing.T) {
require.NoError(t, err)

importer := &githubImporter{}
err = importer.Init(ctx, backend, core.Configuration{
err = importer.Init(ctx, backend, "test", core.Configuration{
confKeyOwner: envUser,
confKeyProject: projectName,
confKeyDefaultLogin: login,
Expand Down
2 changes: 1 addition & 1 deletion bridge/github/import.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ type githubImporter struct {
out chan<- core.ImportResult
}

func (gi *githubImporter) Init(_ context.Context, repo *cache.RepoCache, conf core.Configuration) error {
func (gi *githubImporter) Init(_ context.Context, repo *cache.RepoCache, name string, conf core.Configuration) error {
gi.conf = conf

creds, err := auth.List(repo,
Expand Down
2 changes: 1 addition & 1 deletion bridge/github/import_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ func Test_Importer(t *testing.T) {
ctx := context.Background()

importer := &githubImporter{}
err = importer.Init(ctx, backend, core.Configuration{
err = importer.Init(ctx, backend, "test", core.Configuration{
confKeyOwner: "MichaelMure",
confKeyProject: "git-bug-test-github-bridge",
confKeyDefaultLogin: login,
Expand Down
2 changes: 1 addition & 1 deletion bridge/gitlab/export.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ type gitlabExporter struct {
}

// Init .
func (ge *gitlabExporter) Init(_ context.Context, repo *cache.RepoCache, conf core.Configuration) error {
func (ge *gitlabExporter) Init(_ context.Context, repo *cache.RepoCache, name string, conf core.Configuration) error {
ge.conf = conf
ge.identityClient = make(map[entity.Id]*gitlab.Client)
ge.cachedOperationIDs = make(map[string]string)
Expand Down
4 changes: 2 additions & 2 deletions bridge/gitlab/export_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ func TestPushPull(t *testing.T) {

// initialize exporter
exporter := &gitlabExporter{}
err = exporter.Init(ctx, backend, core.Configuration{
err = exporter.Init(ctx, backend, "test", core.Configuration{
confKeyProjectID: strconv.Itoa(projectID),
confKeyGitlabBaseUrl: defaultBaseURL,
confKeyDefaultLogin: login,
Expand Down Expand Up @@ -223,7 +223,7 @@ func TestPushPull(t *testing.T) {
require.NoError(t, err)

importer := &gitlabImporter{}
err = importer.Init(ctx, backend, core.Configuration{
err = importer.Init(ctx, backend, "test", core.Configuration{
confKeyProjectID: strconv.Itoa(projectID),
confKeyGitlabBaseUrl: defaultBaseURL,
confKeyDefaultLogin: login,
Expand Down
2 changes: 1 addition & 1 deletion bridge/gitlab/import.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ type gitlabImporter struct {
out chan<- core.ImportResult
}

func (gi *gitlabImporter) Init(_ context.Context, repo *cache.RepoCache, conf core.Configuration) error {
func (gi *gitlabImporter) Init(_ context.Context, repo *cache.RepoCache, name string, conf core.Configuration) error {
gi.conf = conf

creds, err := auth.List(repo,
Expand Down
2 changes: 1 addition & 1 deletion bridge/gitlab/import_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ func TestImport(t *testing.T) {
ctx := context.Background()

importer := &gitlabImporter{}
err = importer.Init(ctx, backend, core.Configuration{
err = importer.Init(ctx, backend, "test", core.Configuration{
confKeyProjectID: projectID,
confKeyGitlabBaseUrl: defaultBaseURL,
confKeyDefaultLogin: login,
Expand Down
4 changes: 4 additions & 0 deletions bridge/jira/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ func (*Jira) ValidParams() map[string]interface{} {
"Login": nil,
"CredPrefix": nil,
"Project": nil,
"Filter": nil,
}
}

Expand All @@ -57,6 +58,8 @@ func (j *Jira) Configure(repo *cache.RepoCache, params core.BridgeParams) (core.
}
}

filter := params.Filter

fmt.Println(credTypeText)
credTypeInput, err := input.PromptChoice("Authentication mechanism", []string{"SESSION", "TOKEN"})
if err != nil {
Expand Down Expand Up @@ -99,6 +102,7 @@ func (j *Jira) Configure(repo *cache.RepoCache, params core.BridgeParams) (core.
conf[core.ConfigKeyTarget] = target
conf[confKeyBaseUrl] = baseURL
conf[confKeyProject] = project
conf[confKeyFilter] = filter
conf[confKeyCredentialType] = credType
conf[confKeyDefaultLogin] = login

Expand Down
18 changes: 10 additions & 8 deletions bridge/jira/export.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ var (

// jiraExporter implement the Exporter interface
type jiraExporter struct {
name string
conf core.Configuration

// cache identities clients
Expand All @@ -44,7 +45,8 @@ type jiraExporter struct {
}

// Init .
func (je *jiraExporter) Init(ctx context.Context, repo *cache.RepoCache, conf core.Configuration) error {
func (je *jiraExporter) Init(ctx context.Context, repo *cache.RepoCache, name string, conf core.Configuration) error {
je.name = name
je.conf = conf
je.identityClient = make(map[entity.Id]*Client)
je.cachedOperationIDs = make(map[entity.Id]string)
Expand Down Expand Up @@ -212,10 +214,10 @@ func (je *jiraExporter) exportBug(ctx context.Context, b *cache.BugCache, out ch

// skip bug if it is a jira bug but is associated with another project
// (one bridge per JIRA project)
project, ok := snapshot.GetCreateMetadata(metaKeyJiraProject)
if ok && !stringInSlice(project, []string{je.project.ID, je.project.Key}) {
bridge, ok := snapshot.GetCreateMetadata(metaKeyJiraBridge)
if ok && je.name != bridge {
out <- core.NewExportNothing(
b.Id(), fmt.Sprintf("issue tagged with project: %s", project))
b.Id(), fmt.Sprintf("issue tagged with project: %s", bridge))
return nil
}

Expand Down Expand Up @@ -273,7 +275,7 @@ func (je *jiraExporter) exportBug(ctx context.Context, b *cache.BugCache, out ch
out <- core.NewExportBug(b.Id())
// mark bug creation operation as exported
err = markOperationAsExported(
b, createOp.Id(), id, je.project.Key, time.Time{})
b, createOp.Id(), id, je.name, time.Time{})
if err != nil {
err := errors.Wrap(err, "marking operation as exported")
out <- core.NewExportError(err, b.Id())
Expand Down Expand Up @@ -437,10 +439,10 @@ func (je *jiraExporter) exportBug(ctx context.Context, b *cache.BugCache, out ch
return nil
}

func markOperationAsExported(b *cache.BugCache, target entity.Id, jiraID, jiraProject string, exportTime time.Time) error {
func markOperationAsExported(b *cache.BugCache, target entity.Id, jiraID, name string, exportTime time.Time) error {
newMetadata := map[string]string{
metaKeyJiraId: jiraID,
metaKeyJiraProject: jiraProject,
metaKeyJiraId: jiraID,
metaKeyJiraBridge: name,
}
if !exportTime.IsZero() {
newMetadata[metaKeyJiraExportTime] = exportTime.Format(http.TimeFormat)
Expand Down
32 changes: 20 additions & 12 deletions bridge/jira/import.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ const (

// jiraImporter implement the Importer interface
type jiraImporter struct {
name string
conf core.Configuration

client *Client
Expand All @@ -32,7 +33,8 @@ type jiraImporter struct {
}

// Init .
func (ji *jiraImporter) Init(ctx context.Context, repo *cache.RepoCache, conf core.Configuration) error {
func (ji *jiraImporter) Init(ctx context.Context, repo *cache.RepoCache, name string, conf core.Configuration) error {
ji.name = name
ji.conf = conf

var cred auth.Credential
Expand Down Expand Up @@ -76,31 +78,37 @@ end:
return err
}

func filterOf(filter string, project string, since time.Time) string {
if filter == "" {
filter = fmt.Sprintf("project = \"%s\"", project)
}
return fmt.Sprintf("%s AND updatedDate>\"%s\"", filter, since.Format("2006-01-02 15:04"))
}

// ImportAll iterate over all the configured repository issues and ensure the
// creation of the missing issues / timeline items / edits / label events ...
func (ji *jiraImporter) ImportAll(ctx context.Context, repo *cache.RepoCache, since time.Time) (<-chan core.ImportResult, error) {
sinceStr := since.Format("2006-01-02 15:04")
project := ji.conf[confKeyProject]
filter := filterOf(ji.conf[confKeyFilter], ji.conf[confKeyProject], since)

out := make(chan core.ImportResult)
ji.out = out

go func() {
defer close(ji.out)

message, err := ji.client.Search(
fmt.Sprintf("project=%s AND updatedDate>\"%s\"", project, sinceStr), 0, 0)
fmt.Printf("Searchin for %s\n", filter)

message, err := ji.client.Search(filter, 0, 0)
if err != nil {
out <- core.NewImportError(err, "")
return
}

fmt.Printf("So far so good. Have %d issues to import\n", message.Total)

jql := fmt.Sprintf("project=%s AND updatedDate>\"%s\"", project, sinceStr)
var searchIter *SearchIterator
for searchIter =
ji.client.IterSearch(jql, defaultPageSize); searchIter.HasNext(); {
ji.client.IterSearch(filter, defaultPageSize); searchIter.HasNext(); {
issue := searchIter.Next()
b, err := ji.ensureIssue(repo, *issue)
if err != nil {
Expand Down Expand Up @@ -192,9 +200,9 @@ func (ji *jiraImporter) ensurePerson(repo *cache.RepoCache, user User) (*cache.I
}

i, err = repo.NewIdentityRaw(
user.DisplayName,
user.EmailAddress,
user.Key,
strings.TrimSpace(user.DisplayName),
strings.TrimSpace(user.EmailAddress),
strings.TrimSpace(user.Key),
"",
map[string]string{
metaKeyJiraUser: user.Key,
Expand Down Expand Up @@ -224,7 +232,7 @@ func (ji *jiraImporter) ensureIssue(repo *cache.RepoCache, issue Issue) (*cache.

return excerpt.CreateMetadata[core.MetaKeyOrigin] == target &&
excerpt.CreateMetadata[metaKeyJiraId] == issue.ID &&
excerpt.CreateMetadata[metaKeyJiraProject] == ji.conf[confKeyProject]
excerpt.CreateMetadata[metaKeyJiraBridge] == ji.name
})
if err != nil && err != bug.ErrBugNotExist {
return nil, err
Expand All @@ -249,7 +257,7 @@ func (ji *jiraImporter) ensureIssue(repo *cache.RepoCache, issue Issue) (*cache.
core.MetaKeyOrigin: target,
metaKeyJiraId: issue.ID,
metaKeyJiraKey: issue.Key,
metaKeyJiraProject: ji.conf[confKeyProject],
metaKeyJiraBridge: ji.name,
metaKeyJiraBaseUrl: ji.conf[confKeyBaseUrl],
})
if err != nil {
Expand Down
3 changes: 2 additions & 1 deletion bridge/jira/jira.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,14 @@ const (
metaKeyJiraDerivedId = "jira-derived-id"
metaKeyJiraKey = "jira-key"
metaKeyJiraUser = "jira-user"
metaKeyJiraProject = "jira-project"
metaKeyJiraBridge = "jira-bridge"

@MichaelMure MichaelMure Nov 28, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a breaking change, it's going to break the import of people already using that bridge.

@nxmatic nxmatic Nov 29, 2020

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure, I've planned to use the bridge as the selector. ie: the bridge that import should do the export. I think it's a more strict this way. This is because I'm replacing the project constraint with the filter if defined.

@nxmatic nxmatic Nov 29, 2020

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can have a back-ward mechanism, will preserve the project meta key and use it if there is no jira bridge key defined. The jira bridge meta key will be only used when the user will have provided a filter overriding the project filter.

metaKeyJiraBaseUrl = "jira-base-url"
metaKeyJiraExportTime = "jira-export-time"
metaKeyJiraLogin = "jira-login"

confKeyBaseUrl = "base-url"
confKeyProject = "project"
confKeyFilter = "filter"
confKeyDefaultLogin = "default-login"
confKeyCredentialType = "credentials-type" // "SESSION" or "TOKEN"
confKeyIDMap = "bug-id-map"
Expand Down
2 changes: 1 addition & 1 deletion bridge/launchpad/import.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ type launchpadImporter struct {
conf core.Configuration
}

func (li *launchpadImporter) Init(_ context.Context, repo *cache.RepoCache, conf core.Configuration) error {
func (li *launchpadImporter) Init(_ context.Context, repo *cache.RepoCache, name string, conf core.Configuration) error {
li.conf = conf
return nil
}
Expand Down
1 change: 1 addition & 0 deletions commands/bridge_configure.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ git bug bridge configure \
flags.BoolVar(&options.tokenStdin, "token-stdin", false, "Will read the token from stdin and ignore --token")
flags.StringVarP(&options.params.Owner, "owner", "o", "", "The owner of the remote repository")
flags.StringVarP(&options.params.Project, "project", "p", "", "The name of the remote repository")
flags.StringVarP(&options.params.Filter, "filter", "f", "", "The issues filter")

return cmd
}
Expand Down
4 changes: 4 additions & 0 deletions doc/man/git-bug-bridge-configure.1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,10 @@ Configure a new bridge by passing flags or/and using interactive terminal prompt
\fB\-p\fP, \fB\-\-project\fP=""
The name of the remote repository

.PP
\fB\-f\fP, \fB\-\-filter\fP=""
The issues filter

.PP
\fB\-h\fP, \fB\-\-help\fP[=false]
help for configure
Expand Down
1 change: 1 addition & 0 deletions doc/md/git-bug_bridge_configure.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ git bug bridge configure \
--token-stdin Will read the token from stdin and ignore --token
-o, --owner string The owner of the remote repository
-p, --project string The name of the remote repository
-f, --filter string The issues filter
-h, --help help for configure
```

Expand Down
Loading