gssapi: delete half-built security context so auth can continue #5236

jonathanturcotte · 2019-09-20T17:06:15Z

As per the GNU Generic Security Service (GSS) API Reference Manual on page 15:

If the initial call of gss_init_sec_context() fails, the implementation should not create a context object, and should leave the value of the context_handle parameter set to GSS_C_NO_CONTEXT to indicate this. In the event of a failure on a subsequent call, the implementation is permitted to delete the "half-built" security context (in which case it should set the context_handle parameter to GSS_C_NO_CONTEXT), but the preferred behavior is to leave the security context untouched for the application to delete (using gss_delete_sec_context).

I have tested and confirmed that this fixes the issue we were having with our bitbucket + kerberos setup. Running the online tests against our bitbucket setup now succeeds:

[centos@rhel7 build]$ kinit root/admin
Password for root/admin@KXLABS.COM:
[centos@rhel7 build]$ export GITTEST_REMOTE_URL=https://bitbucket.kxlabs.com:8080/scm/~admin/libgit2_repo.git
[centos@rhel7 build]$ export GITTEST_REMOTE_DEFAULT=ok
[centos@rhel7 build]$ ./libgit2_clar -ionline -sonline
Loaded 381 suites:
Started (test status codes: OK='.' FAILURE='F' SKIPPED='S')

online::badssl...S
online::clone............S.SSSSS...SSS.S.
online::fetch...........
online::fetchhead......
online::push.....................
online::remotes......

Note that branched off of e24b885, as recent changes seem to have broken the -DUSE_GSSAPI=ON build.

ethomson · 2019-09-20T17:25:08Z

Thanks for the PR. I have two concerns:

What that document doesn't say is that the application should also retry in this case. I think that you're right and it's clear that we should be, but I'd like to see some guidance or have some better understanding as to why this is the right thing to do here. Even an example where other code is also doing this would be helpful.
I want to walk through and make sure that we won't accidentally end up in an infinite loop here because we're converting this failure case into a continue. The retry logic should be at a higher level after the HTTP refactoring, but I just want to double-check.

jonathanturcotte · 2019-09-20T20:59:58Z

Those are very valid points, and unfortunately I'm not very familiar with gssapi :(

Curl seems to have a case where they also restart authentication, though some of the finer details elude me:
https://github.com/curl/curl/blob/7e513c1048fc506885ded033a37151309cc71af7/lib/http_negotiate.c#L80-L98

Curl_http_auth_cleanup_negotiate calls Curl_auth_cleanup_spnego, which resets the security context like we do:
https://github.com/curl/curl/blob/7e513c1048fc506885ded033a37151309cc71af7/lib/vauth/spnego_gssapi.c#L254-L258

ethomson · 2019-09-22T20:35:37Z

I set up a server that speaks SPNEGO to walk through this and I'm seeing some other problems in my configuration that I'm still trying to understand.

jonathanturcotte · 2019-09-26T14:22:29Z

Let me know if there's anything I can do to help move this forward.

And thanks for the help so far! I gave #5238 a quick test on my mac and it worked great.

mattmaynes2 · 2019-10-07T14:24:44Z

We are hitting the same issue as @jonathanturcotte seems to have fixed with this PR. Are there any plans to move forward on this issue @ethomson?

jonathanturcotte · 2019-10-17T19:34:29Z

@ethomson: sorry to bump this again, just wondering if there was any updates or you needed anything from me to help move this forward

ethomson · 2019-10-18T10:40:31Z

@ethomson: sorry to bump this again, just wondering if there was any updates or you needed anything from me to help move this forward

Sorry for the delay @jonathanturcotte, it seems like I definitely broke gssapi more soundly when I refactored the networking. Even your PR is not enough to fix my system :( I'm digging in but it'll be a little bit longer before I understand what we've done wrong. Hopefully I'll know more over the weekend.

ethomson · 2019-10-21T10:36:02Z

Aha. I think I'm starting to understand: are you using an Apache (or maybe nginx?) web server?

What (I think) I'm seeing (talking to an Apache server) is that it's tearing down the authenticated connection, which is absolutely insane, but we've seen reports of before (though with NTLM authentication). This is really not how the NTLM and SPNEGO authentication mechanisms were designed, doesn't really follow the spec and is insanely poorly performing. But something that we definitely need to support independent of all those things.

jonathanturcotte · 2019-10-21T15:12:21Z

We're currently using Bitbucket (which uses Tomcat), and this sounds like what I've seen debugging this on my end.

We have also tried to do SPNEGO with an Apache setup, and we were seeing similar unexplained failures at times. I think that behaviour that you're describing is probably the root cause of both.

ethomson · 2019-10-21T15:15:59Z

Great. I've pulled in your change but I'll have some more for you to test in your environment 🔜.

jonathanturcotte · 2019-10-21T15:28:21Z

Awesome, thanks for your help with this!

jonathanturcotte · 2019-11-19T17:24:58Z

Any news @ethomson? This is unfortunately still preventing us from using Git.

ethomson · 2019-11-19T22:12:21Z

I added (opt-in) expect/continue support in #5286

I'm not done working on this, but I'd encourage you to ship a private branch with the changes that you need to get unblocked in the meantime.

jonathanturcotte · 2019-11-26T20:00:39Z

I've given that branch a try and now I'm seeing "no Content-Type header in response". For some extra context on what Bitbucket is giving us, here is the output of a GIT_CURL_VERBOSE=1 git clone on our repository:

centos@rhel7:~ $ GIT_CURL_VERBOSE=1 git clone https://bitbucket.kxlabs.com:8080/kerberos-scm/test/test.git                                                                                                                                                 
Cloning into 'test'...
* Couldn't find host bitbucket.kxlabs.com in the .netrc file; using defaults
* About to connect() to bitbucket.kxlabs.com port 8080 (#0)
*   Trying 35.183.0.112...
* Connected to bitbucket.kxlabs.com (35.183.0.112) port 8080 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=bitbucket.kxlabs.com,OU=kxlabs,O=kx,L=Ottawa,ST=Ontario,C=CA
*       start date: Sep 18 16:26:48 2019 GMT
*       expire date: Dec 17 16:26:48 2019 GMT
*       common name: bitbucket.kxlabs.com
*       issuer: CN=bitbucket.kxlabs.com,OU=kxlabs,O=kx,L=Ottawa,ST=Ontario,C=CA
> GET /kerberos-scm/test/test.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/2.23.0
Host: bitbucket.kxlabs.com:8080
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en-CA, *;q=0.9
Pragma: no-cache

< HTTP/1.1 401
< X-AREQUESTID: @1P4OJQCx1187x236x0
< X-ASEN: SEN-L14608078
< WWW-Authenticate: Basic realm="Atlassian Bitbucket"
< WWW-Authenticate: Negotiate
< Transfer-Encoding: chunked
< Date: Tue, 26 Nov 2019 19:47:44 GMT
<
* Ignoring the response-body
* Connection #0 to host bitbucket.kxlabs.com left intact
* Issue another request to this URL: 'https://bitbucket.kxlabs.com:8080/kerberos-scm/test/test.git/info/refs?service=git-upload-pack'
* Couldn't find host bitbucket.kxlabs.com in the .netrc file; using defaults
* Found bundle for host bitbucket.kxlabs.com: 0x2663b50
* Re-using existing connection! (#0) with host bitbucket.kxlabs.com
* Connected to bitbucket.kxlabs.com (35.183.0.112) port 8080 (#0)
> GET /kerberos-scm/test/test.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/2.23.0
Host: bitbucket.kxlabs.com:8080
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en-CA, *;q=0.9
Pragma: no-cache

< HTTP/1.1 401
< X-AREQUESTID: @1P4OJQCx1187x237x0
< X-ASEN: SEN-L14608078
< WWW-Authenticate: Basic realm="Atlassian Bitbucket"
< WWW-Authenticate: Negotiate
< Transfer-Encoding: chunked
< Date: Tue, 26 Nov 2019 19:47:44 GMT
<
* Ignoring the response-body
* Connection #0 to host bitbucket.kxlabs.com left intact
* Issue another request to this URL: 'https://bitbucket.kxlabs.com:8080/kerberos-scm/test/test.git/info/refs?service=git-upload-pack'
* Couldn't find host bitbucket.kxlabs.com in the .netrc file; using defaults
* Found bundle for host bitbucket.kxlabs.com: 0x2663b50
* Re-using existing connection! (#0) with host bitbucket.kxlabs.com
* Connected to bitbucket.kxlabs.com (35.183.0.112) port 8080 (#0)
* Server auth using GSS-Negotiate with user ''
> GET /kerberos-scm/test/test.git/info/refs?service=git-upload-pack HTTP/1.1
Authorization: Negotiate [TOKEN HERE]

User-Agent: git/2.23.0
Host: bitbucket.kxlabs.com:8080
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en-CA, *;q=0.9
Pragma: no-cache

< HTTP/1.1 200
< X-AREQUESTID: @1P4OJQCx1187x238x0
< X-ASEN: SEN-L14608078
< WWW-Authenticate: Basic realm="Atlassian Bitbucket"
< WWW-Authenticate: Negotiate
< Transfer-Encoding: chunked
< Date: Tue, 26 Nov 2019 19:47:44 GMT
<
* Closing connection 0
* Couldn't find host bitbucket.kxlabs.com in the .netrc file; using defaults
* About to connect() to bitbucket.kxlabs.com port 8080 (#1)
*   Trying 35.183.0.112...
* Connected to bitbucket.kxlabs.com (35.183.0.112) port 8080 (#1)
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=bitbucket.kxlabs.com,OU=kxlabs,O=kx,L=Ottawa,ST=Ontario,C=CA
*       start date: Sep 18 16:26:48 2019 GMT
*       expire date: Dec 17 16:26:48 2019 GMT
*       common name: bitbucket.kxlabs.com
*       issuer: CN=bitbucket.kxlabs.com,OU=kxlabs,O=kx,L=Ottawa,ST=Ontario,C=CA
> GET /kerberos-scm/test/test.git/HEAD HTTP/1.1
User-Agent: git/2.23.0
Host: bitbucket.kxlabs.com:8080
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en-CA, *;q=0.9
Pragma: no-cache

< HTTP/1.1 401
< X-AREQUESTID: @1P4OJQCx1187x239x0
< X-ASEN: SEN-L14608078
< WWW-Authenticate: Basic realm="Atlassian Bitbucket"
< WWW-Authenticate: Negotiate
< Transfer-Encoding: chunked
< Date: Tue, 26 Nov 2019 19:47:44 GMT
<
* Ignoring the response-body
* Connection #1 to host bitbucket.kxlabs.com left intact
* Issue another request to this URL: 'https://bitbucket.kxlabs.com:8080/kerberos-scm/test/test.git/HEAD'
* Couldn't find host bitbucket.kxlabs.com in the .netrc file; using defaults
* Found bundle for host bitbucket.kxlabs.com: 0x2709420
* Re-using existing connection! (#1) with host bitbucket.kxlabs.com
* Connected to bitbucket.kxlabs.com (35.183.0.112) port 8080 (#1)
* Server auth using GSS-Negotiate with user ''
> GET /kerberos-scm/test/test.git/HEAD HTTP/1.1
Authorization: Negotiate [TOKEN HERE]
User-Agent: git/2.23.0
Host: bitbucket.kxlabs.com:8080
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en-CA, *;q=0.9
Pragma: no-cache

< HTTP/1.1 200
< X-AREQUESTID: @1P4OJQCx1187x240x0
< X-ASEN: SEN-L14608078
< WWW-Authenticate: Basic realm="Atlassian Bitbucket"
< WWW-Authenticate: Negotiate
< Transfer-Encoding: chunked
< Date: Tue, 26 Nov 2019 19:47:44 GMT
<
* Closing connection 1
warning: You appear to have cloned an empty repository.

ethomson · 2019-11-26T23:15:54Z

I appreciate the extra context. This is very surprising - bitbucket is speaking "dumb" HTTP here. Is it possible that the git version on the client or server is very old?

https://confluence.atlassian.com/bitbucketserverkb/git-commands-return-error-code-400-779171756.html

jonathanturcotte · 2019-11-27T16:46:04Z

Ah, figured it out. It was a misconfiguration of our kerberos plugin on the server side, causing a failure to authenticate. Weird how commandline git creates an empty repository anyways 😒 This branch is now working great with GSSAPI repositories though!

Unfortunately however I get segfaults in libssl when cloning over https 😩 SSH and Negotiate over HTTP work fine. We had this code working with 28.2, but I have a feeling that maybe something has changed in this branch and we need to change how we're calling the api. Here is the backtrace, if you have any ideas on what to check first that would be really helpful!

Program received signal SIGSEGV, Segmentation fault.
0x00007f5ca4f7d537 in ssl3_read_bytes () from /lib64/libssl.so
(gdb) backtrace
#0  0x00007f5ca4f7d537 in ssl3_read_bytes () from /lib64/libssl.so
#1  0x00007f5ca4f796d4 in ssl3_read_internal () from /lib64/libssl.so
#2  0x00007f5ca301877c in openssl_read (stream=0x1761e90, data=0x178cfae, len=2147483647)
    at /home/centos/libgit2-gssapi/src/streams/openssl.c:663
#3  0x00007f5ca2fcd856 in git_stream_read (st=0x1761e90, data=0x178cfae, len=2147483647)
    at /home/centos/libgit2-gssapi/src/stream.h:50
#4  0x00007f5ca2fcd955 in recv_stream (buf=0x177cf88) at /home/centos/libgit2-gssapi/src/netops.c:45
#5  0x00007f5ca2fcd875 in gitno_recv (buf=0x177cf88) at /home/centos/libgit2-gssapi/src/netops.c:20
#6  0x00007f5ca30292a6 in http_stream_read (stream=0x17a2550,
    buffer=0x176cded "ead\n004215adc3bc386735fb511ce00d49bf6996fd45f5ed refs/", buf_size=65475, bytes_read=0x7ffe9e441660)
    at /home/centos/libgit2-gssapi/src/transports/http.c:1240
#7  0x00007f5ca302c0f5 in git_smart__recv_cb (buf=0x176cd88) at /home/centos/libgit2-gssapi/src/transports/smart.c:25
#8  0x00007f5ca2fcd875 in gitno_recv (buf=0x176cd88) at /home/centos/libgit2-gssapi/src/netops.c:20
#9  0x00007f5ca302ef82 in git_smart__store_refs (t=0x176cbf0, flushes=2)
---Type <return> to continue, or q <return> to quit---
    at /home/centos/libgit2-gssapi/src/transports/smart_protocol.c:56
#10 0x00007f5ca302c8f6 in git_smart__connect (transport=0x176cbf0, url=0x16ecfd0 "https://github.com/libgit2/libgit2",
    cred_acquire_cb=0x7f5ca32afb60 <test_connection_cb>, cred_acquire_payload=0x7ffe9e441890, proxy=0x0, direction=0, flags=0)
    at /home/centos/libgit2-gssapi/src/transports/smart.c:255
#11 0x00007f5ca3001293 in git_remote__connect (remote=0x16a1be0, direction=GIT_DIRECTION_FETCH, callbacks=0x7ffe9e4418c0,
    conn=0x7ffe9e441860) at /home/centos/libgit2-gssapi/src/remote.c:747
#12 0x00007f5ca300134e in git_remote_connect (remote=0x16a1be0, direction=GIT_DIRECTION_FETCH, callbacks=0x7ffe9e4418c0, proxy=0x0,
    custom_headers=0x0) at /home/centos/libgit2-gssapi/src/remote.c:776
#13 0x00007f5ca32afec1 in k_libgit2_test_connection (krepo=<optimized out>, kremote=<optimized out>, kuser=0x7f5ca5402230,
    kpass=<optimized out>) at src/q_git_remote.c:180

ethomson · 2019-11-27T22:49:20Z

Weird how commandline git creates an empty repository anyways

This is a function of git's ability to speak both "dumb" http and "smart" http protocols. Dumb HTTP is basically a web server pointed at a .git folder. It's old and rather broken; libgit2 doesn't support it, but that's why you get different behavior between the two.

#3 0x00007f5ca2fcd856 in git_stream_read (st=0x1761e90, data=0x178cfae, len=2147483647)
at /home/centos/libgit2-gssapi/src/stream.h:50

Looks like I broke something, that's definitely not right. I'll take a look.

ethomson · 2019-11-27T22:52:58Z

Which commit are you using for testing, @jonathanturcotte? Were you using the expect/continue changes or the most recent commit without enabling expect/continue? From the stack trace, it looks like I've done some more refactoring here than what you're running.

jonathanturcotte · 2019-11-28T16:27:31Z

Looks like you are right, I was testing 2a171b4. I've rebuilt to the latest commit on that branch, but am still seeing the segfault :/ The backtrace does look slightly different now:

Program received signal SIGSEGV, Segmentation fault.
0x00007f4e4ea64537 in ssl3_read_bytes () from /lib64/libssl.so
(gdb) backtrace
#0  0x00007f4e4ea64537 in ssl3_read_bytes () from /lib64/libssl.so
#1  0x00007f4e4ea606d4 in ssl3_read_internal () from /lib64/libssl.so
#2  0x00007f4e4cafeacf in openssl_read (stream=0x971f00, data=0x99d050, len=2147483647)
    at /home/centos/libgit2-gssapi/src/streams/openssl.c:679
#3  0x00007f4e4cab39f3 in git_stream_read (st=0x971f00, data=0x99d050, len=2147483647)
    at /home/centos/libgit2-gssapi/src/stream.h:50
#4  0x00007f4e4cab3af2 in recv_stream (buf=0x98cff8) at /home/centos/libgit2-gssapi/src/netops.c:45
#5  0x00007f4e4cab3a12 in gitno_recv (buf=0x98cff8) at /home/centos/libgit2-gssapi/src/netops.c:20
#6  0x00007f4e4cb0ecf6 in read_response (bytes_read=0x7fff30437ce0, auth_replay=0x7fff30437c8f, s=0x9b25c0, t=0x98ce30,
    parser_settings=0x98cfc0, body_buffer=0x97ce5f "ge\n00416e286e8dc59874db30b6fbb0ca5d32d4a2b5642c refs/pull/1526/hea",
    body_buf_size=65473) at /home/centos/libgit2-gssapi/src/transports/http.c:968
#7  0x00007f4e4cb0f76d in http_stream_read (stream=0x9b25c0,
    buffer=0x97ce5f "ge\n00416e286e8dc59874db30b6fbb0ca5d32d4a2b5642c refs/pull/1526/hea", buf_size=65473, bytes_read=0x7fff30437ce0)
    at /home/centos/libgit2-gssapi/src/transports/http.c:1279
#8  0x00007f4e4cb12628 in git_smart__recv_cb (buf=0x97cdf8) at /home/centos/libgit2-gssapi/src/transports/smart.c:25
#9  0x00007f4e4cab3a12 in gitno_recv (buf=0x97cdf8) at /home/centos/libgit2-gssapi/src/netops.c:20
#10 0x00007f4e4cb154b5 in git_smart__store_refs (t=0x97cc60, flushes=2)
    at /home/centos/libgit2-gssapi/src/transports/smart_protocol.c:56
#11 0x00007f4e4cb12e29 in git_smart__connect (transport=0x97cc60, url=0x8fd050 "https://github.com/libgit2/libgit2",
    cred_acquire_cb=0x7f4e4cd96b60 <test_connection_cb>, cred_acquire_payload=0x7fff30437f10, proxy=0x0, direction=0, flags=0)
    at /home/centos/libgit2-gssapi/src/transports/smart.c:255
#12 0x00007f4e4cae7675 in git_remote__connect (remote=0x8b1c60, direction=GIT_DIRECTION_FETCH, callbacks=0x7fff30437f40,
    conn=0x7fff30437ee0) at /home/centos/libgit2-gssapi/src/remote.c:747
#13 0x00007f4e4cae7730 in git_remote_connect (remote=0x8b1c60, direction=GIT_DIRECTION_FETCH, callbacks=0x7fff30437f40, proxy=0x0,
    custom_headers=0x0) at /home/centos/libgit2-gssapi/src/remote.c:776
#14 0x00007f4e4cd96ec1 in k_libgit2_test_connection (krepo=<optimized out>, kremote=<optimized out>, kuser=0x7f4e4ef02230,
    kpass=<optimized out>) at src/q_git_remote.c:180
#15 0x0000000000485e25 in ?? ()

ethomson · 2019-11-28T21:27:11Z

Did you have expect continue support enabled via the option or not?

jonathanturcotte · 2019-11-28T21:51:20Z

Sorry, I'm running this on your branch having compiled it with the -DUSE_GSSAPI=ON option.

ethomson · 2020-01-25T00:47:39Z

Closed via #5286 - thanks again for the patch!

gssapi: delete half-built security context so auth can continue

3340727

ethomson mentioned this pull request Jan 1, 2020

HTTP: Support Apache-based servers with Negotiate #5286

Merged

ethomson closed this Jan 25, 2020

mattmaynes mentioned this pull request Feb 6, 2020

GSSAPI fails with "could not restart authentication" when pulling from Kerberos enabled bitbucket server #5233

Open

gssapi: delete half-built security context so auth can continue #5236

gssapi: delete half-built security context so auth can continue #5236

Uh oh!

Conversation

jonathanturcotte commented Sep 20, 2019

Uh oh!

ethomson commented Sep 20, 2019

Uh oh!

jonathanturcotte commented Sep 20, 2019

Uh oh!

ethomson commented Sep 22, 2019

Uh oh!

jonathanturcotte commented Sep 26, 2019

Uh oh!

mattmaynes2 commented Oct 7, 2019

Uh oh!

jonathanturcotte commented Oct 17, 2019

Uh oh!

ethomson commented Oct 18, 2019

Uh oh!

ethomson commented Oct 21, 2019

Uh oh!

jonathanturcotte commented Oct 21, 2019

Uh oh!

ethomson commented Oct 21, 2019

Uh oh!

jonathanturcotte commented Oct 21, 2019

Uh oh!

jonathanturcotte commented Nov 19, 2019

Uh oh!

ethomson commented Nov 19, 2019

Uh oh!

jonathanturcotte commented Nov 26, 2019

Uh oh!

ethomson commented Nov 26, 2019

Uh oh!

jonathanturcotte commented Nov 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ethomson commented Nov 27, 2019

Uh oh!

ethomson commented Nov 27, 2019

Uh oh!

jonathanturcotte commented Nov 28, 2019

Uh oh!

ethomson commented Nov 28, 2019

Uh oh!

jonathanturcotte commented Nov 28, 2019

Uh oh!

ethomson commented Jan 25, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jonathanturcotte commented Nov 27, 2019 •

edited

Loading