Build: support browserstack test #4803

ygj6 · 2020-10-27T08:39:13Z

Summary

This week I tried to do something for the jQuery test migration, as follows:

I applied for a free browserstack account, and added browserstack configuration to my fork project. It seems that the jQuery test cases run properly in the real browser (Windows 10 Chrome) provided by the browserstack platform.

Local operation steps:

git clone -b browserstack https://github.com/jquery/jquery.git
export BROWSERSTACK_USERNAME=<your browserstack username>
export BROWSERSTACK_ACCESS_KEY=<your browserstack access key>
Run npm run test:browserstack

output:

       Running "karma:browserstack" (karma) task
       27 10 2020 14:21:24.584:INFO [karma-server]: Karma v5.1.0 server started at http://0.0.0.0:9876/
       27 10 2020 14:21:24.586:INFO [launcher]: Launching browsers bs_chrome_win10 with concurrency 3
       27 10 2020 14:21:24.655:INFO [launcher]: Starting browser chrome 86.0 (Windows 10) on BrowserStack
       27 10 2020 14:22:01.426:INFO [launcher.browserstack]: chrome 86.0 (Windows 10) session at https://automate.browserstack.com/builds/f5e04e82b7db52fab492e6147fb7ce6f44c8e337/sessions/8a87ce5a344ff150f8072beeb0be5d32606ca2fc
       27 10 2020 14:22:07.500:INFO [Chrome 86.0.4240.75 (Windows 10)]: Connected on socket Of1EpjUUAKiRdu4CAAAA with id 43017044
       Chrome 86.0.4240.75 (Windows 10): Executed 435 of 1094 SUCCESS (0 secs / 52.279 secs)
       Chrome 86.0.4240.75 (Windows 10): Executed 493 of 1094 SUCCESS (0 secs / 1 min 2.398 secs)
       Chrome 86.0.4240.75 (Windows 10): Executed 650 of 1094 SUCCESS (0 secs / 1 min 27.918 secs)
       Chrome 86.0.4240.75 (Windows 10): Executed 711 of 1094 SUCCESS (0 secs / 1 min 37.87 secs)
       Chrome 86.0.4240.75 (Windows 10): Executed 1094 of 1094 SUCCESS (4 mins 39.082 secs / 4 mins 2.227 secs)
       TOTAL: 1094 SUCCESS
       
       Done.

Next, I plan to continue to do it, such as integrating into github actions, configuring and running more real browsers, etc.;

I look forward to your guidance and suggestions.

Checklist

All authors have signed the CLA at https://cla.js.foundation/jquery/jquery
New tests have been added to show the fix or feature works
Grunt build and unit tests pass locally with these changes
If needed, a docs issue/PR was created at https://github.com/jquery/api.jquery.com

mgol

Thanks for the PR!

We need to be able to pass a specific browser to BrowserStack as well as the latest & previous version, etc. We'll probably want something like https://github.com/jquery/sizzle/blob/master/test/karma/launchers.js defined so that we don't need to inline the launchers in Gruntfile and we need to be able to pass different browsers somehow.

ygj6 · 2020-12-19T10:28:37Z

Refer to sizzle I made some changes:

Extract the karma configuration from Gruntfile.js to karma.conf.js.
Add browserstack browser configuration.
Register the karma-tests task to support running the browserstack test.

However, I encountered some problems when running the test. For example, some test cases failed to be executed during the test using the Firefox browser. I do not know whether the problem is a case problem or a configuration problem. Can you @mgol confirm it?
https://travis-ci.org/github/ygj6/jquery/jobs/750532994

mgol · 2020-12-21T20:35:10Z

Have a look at the Travis report:
https://travis-ci.com/github/jquery/jquery/jobs/462903353
The browser tests there have run on old versions of Chrome/Firefox: Chrome 83 (latest is 87), Firefox 63 (latest is 84!). That's why they failed.

Please have a look at our browser support matrix: https://jquery.com/browser-support/. We only want to run tests on these browsers.

ygj6 · 2020-12-22T02:17:02Z

We only want to run tests on these browsers.

@mgol Yes, I know. I'm asking why Firefox 78 (ESR) and Firefox 83 fail in browserstack.

You can check the Travis report on ygj6/jquery:https://travis-ci.org/github/ygj6/jquery/jobs/750532994, This is run in the browserstack.

Travis report on browserstack:

Firefox 83.0 (Windows 10) manipulation jQuery.append with crossorigin attribute FAILED
	Origin header sent
	Expected: true
	Actual: false
	window.corsCallback@test/unit/manipulation.js:2311:10
	@test/data/mock.php=script&cors=1&callback=corsCallback:1:13

Firefox 78.0 (Windows 7) manipulation jQuery.append with crossorigin attribute FAILED
	Origin header sent
	Expected: true
	Actual: false
	window.corsCallback@test/unit/manipulation.js:2311:10
	@test/data/mock.php=script&cors=1&callback=corsCallback:1:13

mgol · 2020-12-22T10:25:28Z

@ygj6 Where do you see these errors? Travis shows old versions of Chrome/Firefox as I mentioned above & I don't see errors anywhere else.

ygj6 · 2020-12-23T02:49:57Z

Please check this Travis report on my repository ygj6/jquery: https://travis-ci.org/github/ygj6/jquery/jobs/750532994

In this PR, the secure in .travis.yml is generated by my ygj6/jquery repository. Therefore, the secure does not take effect in the jquery/jquery repository, Please replace the secure to the correct secure generated by the jquery/jquery repository.

The way to generate encryption-keys(secure) :
Use travis to generate encryption-keys(secure) containing BROWSER_STACK_USERNAME=*** and BROWSER_STACK_ACCESS_KEY=***.
https://docs.travis-ci.com/user/encryption-keys/#usage

mgol · 2020-12-23T12:23:40Z

@ygj6 I think you can find the reason here:

jquery/test/unit/manipulation.js

Lines 2295 to 2303 in a503c69

    
           // We need to simulate cross-domain requests with the feature that 
        
           // both 127.0.0.1 and localhost point to the mock http server. 
        
           // Skip the the test if we are not in localhost but make sure we run 
        
           // it in Karma. 
        
           QUnit[ 
        
           	jQuery.ajax && ( window.__karma__ || location.hostname === "localhost" ) ? 
        
           		"test" : 
        
           		"skip" 
        
           ]( "jQuery.append with crossorigin attribute", function( assert ) {

We need localhost & 127.0.0.1 to resolve to the same address on which Karma runs and I guess that's not the case when we use BrowserStack? We might need to skip that test when BrowserStack is used as well.

ygj6 · 2020-12-26T03:05:54Z

I tested and found these:

Browserstack+chrome 87 test scene request header information has origin, browserstack+firefox 83 scene request header information does not have origin;
Both Chrome Headless and firefox headless have origin in the scene request header;
Use my PC Chrome 87 and Firefox 84 test request header information has origin.

Although the request header of browserstack+firefox does not contain origin, but the corsCallback can be successfully called,
it should be crossorigin success. I have checked many documents. I still do not know why the difference exists.
Can we modify the test case, for example, change typeof response.headers.origin === "string" to typeof response.headers.referer === "string", and the test will pass.

jquery/test/unit/manipulation.js

Lines 2310 to 2314 in a503c69

    
           window.corsCallback = function( response ) { 
        
           	assert.ok( typeof response.headers.origin === "string", "Origin header sent" ); 
        
           	window.clearTimeout( timeout ); 
        
           	done(); 
        
           };

We can also use the skip case in the browserstack+firefox scenario.

Browser request header details:

Browserstack + Firefox 83(no origin)

 Object{headers: Object{host: 'bs-local.com:9876', user-agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0', accept: '*/*', accept-language: 'en-US,en;q=0.5', accept-encoding: 'gzip, deflate', connection: 'keep-alive', referer: 'http://bs-local.com:9876/context.html', cookie: 'io=7qIGNAvrEzW9nQD-AAAA'}}

Browserstack + Chrome 87(has origin)

Object{headers: Object{host: 'bs-local.com:9876', origin: 'http://bs-local.com:9876', user-agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36', accept: '/', referer: 'http://bs-local.com:9876/context.html', accept-encoding: 'gzip, deflate', accept-language: 'en-US,en;q=0.9', cookie: 'io=OjuTbAOmXNy5kYj6AAAA', connection: 'keep-alive'}}`

FirefoxHeadless(has origin)

 Object{headers: Object{host: '127.0.0.1:9876', user-agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0', accept: '/', accept-language: 'zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2', accept-encoding: 'gzip, deflate', origin: 'http://localhost:9876', connection: 'keep-alive', referer: 'http://localhost:9876/context.html'}}

ChromeHeadless(has origin)

Object{headers: Object{host: '127.0.0.1:9876', connection: 'keep-alive', origin: 'http://localhost:9876', user-agent:  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/87.0.4280.88 Safari/537.36', accept: '/', sec-fetch-site: 'cross-site', sec-fetch-mode: 'cors', sec-fetch-dest: 'script', referer: 'http://localhost:9876/context.html', accept-encoding: 'gzip, deflate, br', accept-language: 'zh-CN'}}

PC + Firefox 84(has origin)

Object{headers: Object{host: '127.0.0.1:9876', user-agent: 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0', accept: '/', accept-language: 'zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2', accept-encoding: 'gzip, deflate', origin: 'http://localhost:9876', connection: 'keep-alive', referer: 'http://localhost:9876/context.html'}}

PC + Chrome 87(has origin)

Object{headers: Object{host: '127.0.0.1:9876', connection: 'keep-alive', origin: 'http://localhost:9876', user-agent: 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36', accept: '/', sec-fetch-site: 'cross-site', sec-fetch-mode: 'cors', sec-fetch-dest: 'script', referer: 'http://localhost:9876/context.html', accept-encoding: 'gzip, deflate, br', accept-language: 'zh-CN,zh;q=0.9'}}

mgol · 2020-12-26T20:38:04Z

Can we modify the test case, for example, change typeof response.headers.origin === "string" to typeof response.headers.referer === "string", and the test will pass.

We cannot. The whole point of this test is to check whether we got a CORS request which we verify by checking for the Origin header; Referer is always sent so it doesn't work as a replacement.

We should add that in a comment as it may not be clear why we're checking for the Origin header, indeed.

ygj6 · 2021-01-22T02:59:44Z

Now, skipping the cross-domain test in browserstack, all the real browser tests have passed.
https://travis-ci.org/github/ygj6/jquery/jobs/752090159

@mgol Next, do I need to do anything else for this PR?

mgol

@ygj6 some of our tests are flakey so we need a more granular way of restarting jobs that won't re-run all tests in all browsers. In our existing test system, TestSwarm, we can restart each module in each browser, see e.g.: http://swarm.jquery.org/job/11073 (Web Archive link in case it goes down). I'm not sure if we have to be as granular but at least separating browsers would be necessary.

mgol · 2021-04-28T16:36:02Z

test/unit/manipulation.js

 // it in Karma.
 QUnit[
-	jQuery.ajax && ( window.__karma__ || location.hostname === "localhost" ) ?
+	jQuery.ajax && ( ( window.__karma__ && location.hostname !== "bs-local.com" ) || location.hostname === "localhost" ) ?


We want this test running in Karma and this condition creates a risk of us skipping it completely. Can you elaborate? Whatever we need to exclude, we need to do it in a safer way.

mgol · 2021-04-28T16:37:08Z

.travis.yml

        - BROWSERS="FirefoxHeadless"
      addons:
        firefox: latest-esr
+    - name: "Browser tests: browsertack tests"


A typo + different casing:

Suggested change

- name: "Browser tests: browsertack tests"

- name: "Browser tests: BrowserStack tests"

mgol · 2021-09-17T16:18:27Z

Closing & re-opening the PR to trigger the EasyCLA check...

linux-foundation-easycla · 2021-09-17T16:18:33Z

❌ - ygj6 The commit (7591564 ,e1ed6548a571772dd766fe8c9ae4bdc7d5a675f6 ,ed36ee913ca17d7e4ccf30cc587b0ff337038cc3 ,04a4058fc56dd26da02dc031f6b37895aaaf1835) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

mgol · 2021-12-06T23:37:37Z

@ygj6 Hey! It's been a while; are you still interested in finalizing this PR? Your other PR adding GitHub Actions: #4800 has landed, we'll soon remove the Travis setup completely so this needs to be setup in GitHub Actions now. Plus, my other comments need to be addressed.

ygj6 · 2021-12-31T03:46:32Z

It's an honor to contribute to jquery.
Since the PR submission is a little long, I need to spend time sorting out the code.

mgol · 2022-05-16T16:31:13Z

@ygj6 Hey, any updates?

timmywil · 2024-02-16T20:15:01Z

Thanks for your contribution. However, this PR has grown stale and will not be needed soon.

mgol requested changes Dec 1, 2020

View reviewed changes

Build: Support browserstack test

7591564

ygj6 force-pushed the browserstack branch from 4d494c8 to 7591564 Compare December 19, 2020 08:29

Build: add browserstack in travis

e1ed654

ygj6 force-pushed the browserstack branch from 28553f5 to e1ed654 Compare December 19, 2020 09:27

Build: add comments and upgrade the local browser on travis

ed36ee9

Tests: skip cross-domain test on browserstack

04a4058

ygj6 force-pushed the browserstack branch from bb2d123 to 04a4058 Compare December 30, 2020 07:08

Base automatically changed from master to main February 1, 2021 22:02

mgol requested changes Apr 28, 2021

View reviewed changes

mgol closed this Sep 17, 2021

mgol reopened this Sep 17, 2021

mgol mentioned this pull request Jul 13, 2022

Deprecated: Improve $.trim performance for strings with lots of whitespace #5068

Merged

timmywil closed this Feb 16, 2024

github-actions bot locked as resolved and limited conversation to collaborators Sep 24, 2025

	- name: "Browser tests: browsertack tests"
	- name: "Browser tests: BrowserStack tests"

Build: support browserstack test #4803

Build: support browserstack test #4803

Uh oh!

Conversation

ygj6 commented Oct 27, 2020

Summary

Checklist

Uh oh!

mgol left a comment

Choose a reason for hiding this comment

Uh oh!

ygj6 commented Dec 19, 2020

Uh oh!

mgol commented Dec 21, 2020

Uh oh!

ygj6 commented Dec 22, 2020

Uh oh!

mgol commented Dec 22, 2020

Uh oh!

ygj6 commented Dec 23, 2020

Uh oh!

mgol commented Dec 23, 2020

Uh oh!

ygj6 commented Dec 26, 2020

Uh oh!

mgol commented Dec 26, 2020

Uh oh!

ygj6 commented Jan 22, 2021

Uh oh!

mgol left a comment

Choose a reason for hiding this comment

Uh oh!

mgol Apr 28, 2021

Choose a reason for hiding this comment

Uh oh!

mgol Apr 28, 2021

Choose a reason for hiding this comment

Uh oh!

mgol commented Sep 17, 2021

Uh oh!

linux-foundation-easycla bot commented Sep 17, 2021

Uh oh!

mgol commented Dec 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ygj6 commented Dec 31, 2021

Uh oh!

mgol commented May 16, 2022

Uh oh!

timmywil commented Feb 16, 2024

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

mgol commented Dec 6, 2021 •

edited

Loading