Skip to content
This repository was archived by the owner on Jun 8, 2026. It is now read-only.

resolve twingly/twingly-search-api-java#9 - #18

Closed
xSAVIKx wants to merge 5 commits into
twingly:masterfrom
xSAVIKx:betamaxTests
Closed

xSAVIKx wants to merge 5 commits into
twingly:masterfrom
xSAVIKx:betamaxTests

Conversation

@xSAVIKx

@xSAVIKx xSAVIKx commented Mar 12, 2016

Copy link
Copy Markdown
Contributor

added betamax functional tests

added betamax functional tests
Connection: keep-alive
Content-Length: '0'
Host: api.twingly.com
User-Agent: Twingly Search Java Client/null

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happened to notice /null here, is there some problem with reading the version from the property file?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm. I've tried several solutions, so have only checked what's in response only for the first ones (everything was OK there). I'll check what could be wrong.

@walro

walro commented Mar 12, 2016

Copy link
Copy Markdown
Contributor

I guess you noticed the trouble with Java 7 java.lang.UnsupportedClassVersionError for the betamax test class. I took a look at Betamax and it seems like they test with Java 7 so I guess the problem is something else.

@walro

walro commented Mar 12, 2016

Copy link
Copy Markdown
Contributor

Otherwise I think it looks good. How tricky would it be to incorporate this in the Spock/Groovy test suite? :) I understand it might not be worth it, but it would be cool if all tests were using the same test style, if possible.

@xSAVIKx

xSAVIKx commented Mar 12, 2016

Copy link
Copy Markdown
Contributor Author

Ahh, yeah, I've forgotten about java7.
I've already written some notes about java7 to betamax team lead issue 175.
I think that the problem could be in published by betamax Jar-file, I think it's compiled using java8 compiller.

@walro

walro commented Mar 12, 2016

Copy link
Copy Markdown
Contributor

I think that the problem could be in published by betamax Jar-file, I think it's compiled using java8 compiller.

Ah, that makes sense.

xSAVIKx added 2 commits March 12, 2016 14:36
rewrite UrlConnectionClientFunctionTest with groovy and Spock
ensure that VERSION constant is always initialized after version property is gathered from version.properties.
@xSAVIKx

xSAVIKx commented Mar 12, 2016

Copy link
Copy Markdown
Contributor Author

I've rewritten test with groovy and Spock and also changed slightly VERSION constant, so now it will be 100% loaded after version property is gathered from version.properties.

@walro

walro commented Mar 12, 2016

Copy link
Copy Markdown
Contributor

I've rewritten test with groovy and Spock and also changed slightly VERSION constant, so now it will be 100% loaded after version property is gathered from version.properties.

Nice, but it looks like a new error was introduced for Java8. Did it run fine locally for you?

@xSAVIKx

xSAVIKx commented Mar 12, 2016

Copy link
Copy Markdown
Contributor Author

@walro I think it's some Travis-ci bug, cause on my branch tests passed for java 8 (as locally they are also OK).

@xSAVIKx

xSAVIKx commented Mar 14, 2016

Copy link
Copy Markdown
Contributor Author

I think that branch could be only merged into your repo, when betamax will release a new snapshot. (betamaxteam/betamax#186)

betamax-snapshot should be used until new stable release won't be presented.
@xSAVIKx

xSAVIKx commented Mar 14, 2016

Copy link
Copy Markdown
Contributor Author

Again some weird things happened.
After changing betamax to current snapshot my branch is "green" now... But PR build is broken again. For now have no ideas what could be wrong.

@dentarg

dentarg commented Mar 14, 2016

Copy link
Copy Markdown
Contributor

@xSAVIKx we are now using the latest snapshot of betamax, and that snapshot now (since 8 hours ago :) ) changes with every commit to the betamax repo

the only real change that have been made to betamax, that I can spot, since we tried and jdk8 worked, is betamaxteam/betamax#180 (which fixes betamaxteam/betamax#179), could that affect what we are doing here? cc @cowboygneox

@xSAVIKx

xSAVIKx commented Mar 14, 2016

Copy link
Copy Markdown
Contributor Author

@dentarg I'd understand, if my fork is failed also, but, you see, my fork was successfully built. And local build is also OK, so I cannot understand what's wrong.
Probably, while travis is trying to build PR it break smth related to self-signed certificates, that are used to handle https calls. It's the only assumption I can do now.

@dentarg

dentarg commented Mar 14, 2016

Copy link
Copy Markdown
Contributor

Ah that's strange... for the record, here is your green build: https://travis-ci.org/xSAVIKx/twingly-search-api-java/builds/115814335. I restarted our failing build: https://travis-ci.org/twingly/twingly-search-api-java/builds/115814365

@cowboygneox

Copy link
Copy Markdown

@dentarg The change in betamaxteam/betamax#180 will only affect the default configuration, which will be to use the proxy. In most examples and usages, you won't use the default, and thus should have no impact. If you're not configuring the proxy, you're unlikely using Betamax effectively.

@dentarg

dentarg commented Mar 14, 2016

Copy link
Copy Markdown
Contributor

This is really strange, https://travis-ci.org/twingly/twingly-search-api-java/builds/115814365 fails but not
https://travis-ci.org/xSAVIKx/twingly-search-api-java/builds/115814335, but both should be the same code. I'm going to check with Travis CI support before merging.

gradlew check works locally for me (1.7.0_45-b18), so I don't think we have a real problem.

thank you @xSAVIKx for the work here and thank you @cowboygneox for reviving betamax, I will be on the look out for the next relase :)

@dentarg

dentarg commented Mar 14, 2016

Copy link
Copy Markdown
Contributor

@xSAVIKx when I ran locally, I noticed that littleproxy_cert and littleproxy_keystore.jks was created in the project root, perhaps we should add those to .gitignore?

@dentarg

dentarg commented Mar 14, 2016

Copy link
Copy Markdown
Contributor

I think I know the problem with Travis, it doesn't seem to run the latest commit... some sync issue

@dentarg

dentarg commented Mar 14, 2016

Copy link
Copy Markdown
Contributor

I think I know the problem with Travis, it doesn't seem to run the latest commit... some sync issue

No...

@dentarg

dentarg commented Mar 14, 2016

Copy link
Copy Markdown
Contributor

I have now emailed travis ci support

@dentarg

dentarg commented Mar 14, 2016

Copy link
Copy Markdown
Contributor

I also did the merge with master locally and ran the tests, worked fine

@cowboygneox

Copy link
Copy Markdown

@dentarg You should ignore those certs; they are generated as necessary for SSL support. If you commit them, they will likely expire some time in the future causing messy failures. I'll have to add that to the documentation on Betamax.

@xSAVIKx

xSAVIKx commented Mar 14, 2016

Copy link
Copy Markdown
Contributor Author

@xSAVIKx when I ran locally, I noticed that littleproxy_cert and littleproxy_keystore.jks was created in the project root, perhaps we should add those to .gitignore?

@dentarg , as @cowboygneox already mentioned - these are generated certificates to support SSL, so yes, I think I should add them to .gitignore.

add betamax-generated SSL certificated to gitignore
Comment thread build.gradle
cglibVersion = "3.2.0"
objenesisVersion = "2.2"
jaxbVersion = "2.2.11"
betamaxVersion = "2.0.0-alpha-1"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is betamaxVersion used for? Seems to me that only betamaxSnapshotVersion is used.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@walro I think it's left in so we can easily switch back when a proper release of betamax has been made

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I thought it was something like that. I'd like it removed anyway, but we can do it ourselves.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@walro Yeap it was just for simplification of future dependency version change.

@walro

walro commented Mar 15, 2016

Copy link
Copy Markdown
Contributor

@xSAVIKx do you mind trying to restart the tests for your repository? We're trying to figure out the Travis problem. We noticed that you used a slightly older version/build of Betamax (even though it should not matter).

@xSAVIKx

xSAVIKx commented Mar 16, 2016

Copy link
Copy Markdown
Contributor Author

@walro I've just restarted build, so here are the results: build #57

@dentarg

dentarg commented Mar 16, 2016

Copy link
Copy Markdown
Contributor

for "continuous-integration/travis-ci/push" we get WARNING: Playing back from tape ...
for "continuous-integration/travis-ci/pr" we dont' get that

Pretty sure the problem is that betamax isn't using the tape for "continuous-integration/travis-ci/pr", but as to why, I have no idea...

@xSAVIKx

xSAVIKx commented Mar 16, 2016

Copy link
Copy Markdown
Contributor Author

@dentarg betamax tried to use tape, but, I think, smth happened while request was intercepted, cause in PR log, there's INFO:

INFO: loaded tape with 1 recorded interactions from file /home/travis/build/twingly/twingly-search-api-java/src/test/resources/com/twingly/search/client/tapes/search_for_spotify.yaml...
----------------------------------------------------
INFO: loaded tape with 1 recorded interactions from file /home/travis/build/twingly/twingly-search-api-java/src/test/resources/com/twingly/search/client/tapes/search_without_valid_API_key.yaml...

@dentarg

dentarg commented Mar 16, 2016

Copy link
Copy Markdown
Contributor

@xSAVIKx thanks for pointing that out, it's helpful

Old but similar issue: betamaxteam/betamax#134 which says

When the tests is run, it shows:

   INFO: loaded tape with 1 recorded interactions from file piwik_success.yaml...

But the tape is not used in tests. The service makes normal HTTP call.

Sounds a bit like what we are experiencing...

@dentarg

dentarg commented Mar 16, 2016

Copy link
Copy Markdown
Contributor

Also similar betamaxteam/betamax#117

@dentarg

dentarg commented Mar 18, 2016

Copy link
Copy Markdown
Contributor

The response from Travis CI support:

We have investigated this on our side and we are unfortunately a bit lost about what is at play here. It seems like the Betamax filters aren't installed in the PR build. Unfortunately, we don't see how and why this can happen.

I was wondering if it would be possible to add more logs in the Betamax code or even in LittleProxy (where the filters should be installed I believe) to try to understand what's going on. What do you think?

My reply was (which contains some info I haven't shared in this thread yet)

Not exactly the logs you are asking for, but I made the tests run with --debug --full-stacktrace in my own fork:

dentarg#3

jdk8 fail https://travis-ci.org/dentarg/twingly-search-api-java/jobs/116498255
jdk8 pass https://travis-ci.org/dentarg/twingly-search-api-java/jobs/116498139

I will investigate if we can make Betamax and LittleProxy log anything.

Any ideas on what could be causing this? Anything different in the Travis environment for pr vs push?

That was 2 days ago, no further correspondence since then have taken place.

@dentarg

dentarg commented Mar 18, 2016

Copy link
Copy Markdown
Contributor

@cowboygneox sorry to drag you into this again, but do you mind taking a closer look? seems the issue here is similar to some old betamax issues, betamaxteam/betamax#134 and betamaxteam/betamax#117. I think the description "It seems like the Betamax filters aren't installed in the PR build" is accurate, but as to why, I have no clue. Any ideas on what logging could be added our Betamax code, or LittleProxy?

@dentarg

dentarg commented Mar 18, 2016

Copy link
Copy Markdown
Contributor

Hmm... betamaxteam/betamax#193 also fails, and so did betamaxteam/betamax#192 until a race condition was fixed. Related?

@dentarg

dentarg commented Mar 18, 2016

Copy link
Copy Markdown
Contributor

@cowboygneox

Copy link
Copy Markdown

@dentarg It's no fun waking up to an email in the morning that your library is causing others issues :( I'm looking into it right now.

@cowboygneox

Copy link
Copy Markdown

@dentarg Ugh, more SSL issues. These seem to be coming up more frequently now that Java enforces it more. I've actually been struggling myself to get this working (though I have it working) but communicating the steps that I used may be just as challenging. Let me keep looking at your config.

@cowboygneox

Copy link
Copy Markdown

@dentarg (and others observing this): is there any chance you can join the Betamax Gitter so that we can quick round up on this?

@dentarg

dentarg commented Mar 18, 2016

Copy link
Copy Markdown
Contributor

@cowboygneox sure, I can join

@dentarg

dentarg commented Mar 18, 2016

Copy link
Copy Markdown
Contributor

Latest from Travis CI

I had a look at the new log and unfortunately, I could not tell why the Betamax filters don't seem to be installed in the PR build. We are really stumped about how this could be happening and we would really like to understand why it does.

I can give you more details about the differences between a PR build and a push build:

  1. a PR build runs the merge commit instead of the "branch" commit, so there might be more commits than what shows up in the branch. It's not the case here since it's up-to-date with master.

  2. some TRAVIS_… environment variables change, mainly TRAVIS_PULL_REQUEST and the commit range one but as far as I can tell, the build doesn't use any of those env vars. Here's a diff I've generated for the environment variables in each build: https://www.diffchecker.com/ovutx8ah.

  3. encrypted secure env vars are removed if the PR comes from a fork but I don't think there are any secure env vars here, am I right?

Hence, my only other suggestion would be, as I mentioned before, to try instrumenting either or both the Betamax and LittleProxy libraries to better understand why the filters aren't set properly in the PR build. I'm sorry I don't have any easier solution at this time.

Please let me know if this is something you can/want to try and if you need help doing it. We would be happy to assist you doing so.

My reply

Correct, no encrypted secure env vars are in use.

I'm in contact with the current maintainer of Betamax, to debug the problem. My understanding is that he thinks the filters/proxy are used, but there are some problem around SSL certificates (in the pr build), some race condition that are triggered(?)

If you have any ideas, we are both in the Betamax Gitter at https://gitter.im/betamaxteam/betamax

@cowboygneox

Copy link
Copy Markdown

@dentarg I think I solved your issues in the latest 2.0.0-SNAPSHOT of Betamax. Can you try pulling it and see if it works?

@dentarg

dentarg commented Apr 15, 2016

Copy link
Copy Markdown
Contributor

@cowboygneox seems like it stills fails, using https://oss.sonatype.org/content/repositories/snapshots/software/betamax/betamax-junit/2.0.0-SNAPSHOT/betamax-junit-2.0.0-20160415.010203-17.pom, build log at https://travis-ci.org/twingly/twingly-search-api-java/jobs/115858478

Haven't looked into it more than that and I'm afraid I wont be able until some time next week

@cowboygneox

Copy link
Copy Markdown

@dentarg I submitted another SNAPSHOT, and then I also forked your work and ran tests. Your tests passed fine on my run: https://travis-ci.org/cowboygneox/twingly-search-api-java/builds/123307325. Will you kick yours off again?

@dentarg

dentarg commented Apr 15, 2016

Copy link
Copy Markdown
Contributor

@cowboygneox

cowboygneox commented Apr 15, 2016 •

Copy link
Copy Markdown

@dentarg Looks like it failed again. I have absolutely no idea why your branch is failing and mine is consistently passing. All I did was fork your branch and ignore some files. We'll probably have to ask travis-ci to compare.

@cowboygneox

Copy link
Copy Markdown

@dentarg Maybe I need to rebase against master. I'll try that next.

@cowboygneox cowboygneox mentioned this pull request Apr 15, 2016
@cowboygneox

Copy link
Copy Markdown

@dentarg Ok, so I made my own PR #21 and it failed just like this one does, so at least that's consistent. When I run in my own fork, the tests pass, but when we run in a PR, tests fail. I sent an email to travis-ci to have them point out any configuration differences between the types.

@walro

walro commented Jun 4, 2016

Copy link
Copy Markdown
Contributor

It would be nice to finally get this in, @twingly/dev WDYT of the following:

Since the problems we are seeing are likely due to the combination of betamax and https (see #21) we could make the https part configurable for now, defaulting to true. When we record, and run, the Betamax tests we "drop down" to http. Once Betamax can handle https we'll revert the changes.

@cowboygneox

Copy link
Copy Markdown

@walro I am confident that disabling use of https will solve your issues.

Also, VCR doesn't fire up a proxy server, it just stubs out the http_client you're using underneath. It doesn't do a true MITM proxy like we're trying to do. We're trying to support more HTTP clients than we have in the past, and the (seemingly) most appropriate way is just to fire up another server to hit.

@cowboygneox

Copy link
Copy Markdown

@walro Betamax 2.0 official is out. It fixes this issue, but it will require you to change how you run builds on Travis. Please see the documentation on Betamax's readme and let me know if you have any questions.

@walro

walro commented Jul 7, 2016

Copy link
Copy Markdown
Contributor

@walro Betamax 2.0 official is out. It fixes this issue, but it will require you to change how you run builds on Travis. Please see the documentation on Betamax's readme and let me know if you have any questions.

Thanks, on vacation right now. But once back at the office I'll take a look (unless one of my co-workers beats me to it :-))

@dentarg

dentarg commented Jul 7, 2016

Copy link
Copy Markdown
Contributor

I'm also on vacation, but I may take a look at it tonight anyways :-)

On 7 juli 2016, at 19:02, Robin Wallin [email protected] wrote:

@walro Betamax 2.0 official is out. It fixes this issue, but it will require you to change how you run builds on Travis. Please see the documentation on Betamax's readme and let me know if you have any questions.

Thanks, on vacation right now. But once back at the office I'll take a look (unless one of my co-workers beats me to it :-))

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@dentarg dentarg closed this Jul 7, 2016
@dentarg dentarg mentioned this pull request Jul 7, 2016
@dentarg

dentarg commented Jul 7, 2016 •

Copy link
Copy Markdown
Contributor

Closed in favour of #26.

@xSAVIKx
xSAVIKx deleted the betamaxTests branch May 11, 2017 18:09
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants