Skip to content

CF client Invalid auth token error  #569

Closed
orange-cloudfoundry/sec-group-broker-filter
#35

Description

@mohanpindyala

getting following error after a while

reactor.core.Exceptions$BubblingException: org.cloudfoundry.client.v2.CloudFoundryException: CF-InvalidAuthToken(1000): Invalid Auth Token
    at reactor.core.Exceptions.bubble(Exceptions.java:96)
    at reactor.core.publisher.Operators.onErrorDropped(Operators.java:317)
    at reactor.core.publisher.LambdaSubscriber.onError(LambdaSubscriber.java:126)
    at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:193)
    at reactor.core.publisher.MonoFlatMap$FlattenSubscriber.onError(MonoFlatMap.java:166)
    at reactor.core.publisher.MonoCollectList$MonoBufferAllSubscriber.onError(MonoCollectList.java:100)
    at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:134)

any idea

using these versions for java client

<cloudfoundry-client-spring.version>2.0.0.RELEASE</cloudfoundry-client-spring.version>
<cloudfoundry-operations.version>2.0.0.RELEASE</cloudfoundry-operations.version>
<reactor-core.version>3.0.0.RELEASE</reactor-core.version>
<reactor-netty.version>0.5.0.RELEASE</reactor-netty.version>

Activity

  1. cf-gitbot commented on Sep 28, 2016

    @cf-gitbot

    We have created an issue in Pivotal Tracker to manage this:

    https://www.pivotaltracker.com/story/show/131326071

    The labels on this github issue will be updated when the story is started.

  2. twoseat commented on Sep 29, 2016

    @twoseat
    Contributor

    @mohanpindyala I suspect this is because your token has expired. Is 'a while' the same period every time?

  3. nebhale commented on Sep 29, 2016

    @nebhale
    Contributor

    It does certainly appear that the token being presented to the API is invalid. What does your TokenProvider configuration look like?

  4. mohanpindyala commented on Sep 29, 2016

    @mohanpindyala
    Author

    there you go . Also we are using operations object everywhere and not underlying client

    CloudFoundryClient cfClient = ReactorCloudFoundryClient.builder()
        .connectionContext(DefaultConnectionContext.builder()
            .apiHost(cfApiUrl)
            .skipSslValidation(skipSSLValidation)
            .build())
        .tokenProvider(PasswordGrantTokenProvider.builder()
            .password(password)
            .username(username)
            .build())
        .build();
    
    CloudFoundryOperations cfOperations = DefaultCloudFoundryOperations.builder()
        .cloudFoundryClient(cfClient)
        .organization(cforg)
        .space(cfspace)
        .build();
    
  5. nebhale commented on Sep 29, 2016

    @nebhale
    Contributor

    Can you please turn the logger cloudfoundry-client to DEBUG. This should start generating a lot of information about the requests made to the server including negotiation of the token. It'd be helpful to get the contents of these outputs (you can filter out everything else that's application specific) from the start of the application (and the first negotiation of a token) all the way to the moment that the token is not longer accepted.

  6. nebhale commented on Sep 29, 2016

    @nebhale
    Contributor

    In addition to that logging, please update to the latest version (2.0.1.RELEASE) of the client and it's Reactor dependencies. There have been some bug fixes, so let's see if that solves things.

  7. mohanpindyala commented on Sep 29, 2016

    @mohanpindyala
    Author

    oh thats nice , let me update to 2.0.1 release and see if that helps

  8. mohanpindyala commented on Sep 29, 2016

    @mohanpindyala
    Author

    here is the info from the logs . Error happens after roughly 2 hours and this is consistent. This info is from existing code , did not update anything yet. Will update and enable debug mode.

    server started at 2016-09-28 15:02:19.193
    Error happens at 2016-09-28 17:07:34.247

  9. nebhale commented on Nov 2, 2016

    @nebhale
    Contributor

    Since there's been no response on this issue in a couple of weeks, I'm going to close it. If you'd like to see it re-opened, please comment on the issue and I'll reopen it.

  10. added a commit that references this issue on Nov 3, 2016
  11. s-bortolussi commented on Nov 4, 2016

    @s-bortolussi
    Contributor

    @nebhale after upgrading to lastest release 2.0.1.RELEASE, go the same exception.

    Using these versions for java client

            <cf-java-client.version>2.0.1.RELEASE</cf-java-client.version>
            <reactor-core.version>3.0.3.RELEASE</reactor-core.version>
            <reactor-netty.version>0.5.1.RELEASE</reactor-netty.version>
    

    and log activation

    -Dlogging.level.cloudfoundry-client.request=DEBUG -Dlogging.level.cloudfoundry-client.response=DEBUG -Dlogging.level.cloudfoundry-client.token=DEBUG
    

    See following logs including my comments (error seems to be caused by a refresh token expiration.)

    # get an access token
    2016-11-04T15:45:55.09+0100 [APP/PROC/WEB/0]OUT 2016-11-04 14:45:55.096 DEBUG 10 --- [tcp-client-io-1] cloudfoundry-client.request              : GET    https://login.YYYYYYYY:443/oauth/token?client_id=xxxxx&client_secret=&grant_type=password&password=xxxxxxx&username=xxxxxx
    2016-11-04T15:45:55.09+0100 [APP/PROC/WEB/0]OUT 2016-11-04 14:45:55.096 DEBUG 10 --- [tcp-client-io-1] cloudfoundry-client.response             : 200    https://login.YYYYYYYY:443/oauth/token?client_id=xxxxxxx&client_secret=&grant_type=password&password=xxxxxxx&username=xxxxxx
    2016-11-04T15:45:55.10+0100 [APP/PROC/WEB/0]OUT 2016-11-04 14:45:55.099 DEBUG 10 --- [tcp-client-io-1] cloudfoundry-client.token                : JWT Token: xxxxx1
    2016-11-04T15:45:55.12+0100 [APP/PROC/WEB/0]OUT 2016-11-04 14:45:55.124 DEBUG 10 --- [tcp-client-io-1] cloudfoundry-client.request              : GET    https://api.xxxxxxx:443/v2/security_groups?q=namexxxxxx
    2016-11-04T15:45:55.12+0100 [APP/PROC/WEB/0]OUT 2016-11-04 14:45:55.124 DEBUG 10 --- [tcp-client-io-1] cloudfoundry-client.response             : 200    https://api.xxxxxxx:443/v2/security_groups?q=namexxxxx
    2016-11-04T15:45:55.42+0100 [APP/PROC/WEB/0]OUT 2016-11-04 14:45:55.427 DEBUG 10 --- [tcp-client-io-1] cloudfoundry-client.request              : DELETE https://api.xxxxxxx:443/v2/security_groups/xxxxx
    2016-11-04T15:45:55.42+0100 [APP/PROC/WEB/0]OUT 2016-11-04 14:45:55.428 DEBUG 10 --- [tcp-client-io-1] cloudfoundry-client.response             : 204    https://api.xxxxxxx:443/v2/security_groups/xxxxx
    
    # access token expiration, get a new one using refresh token -> SUCCESS
    2016-11-04T15:55:44.18+0100 [APP/PROC/WEB/0]OUT 2016-11-04 14:55:44.185 DEBUG 10 --- [tcp-client-io-2] cloudfoundry-client.request              : GET    https://login.YYYYYYYY:443/oauth/token?grant_type=refresh_token&client_id=xxxxxx&client_secret=&refresh_token=xxx
    2016-11-04T15:55:44.18+0100 [APP/PROC/WEB/0]OUT 2016-11-04 14:55:44.186 DEBUG 10 --- [tcp-client-io-2] cloudfoundry-client.response             : 200    https://login.YYYYYYYY:443/oauth/token?grant_type=refresh_token&client_id=xxxx&client_secret=&refresh_token=xxx
    2016-11-04T15:55:44.18+0100 [APP/PROC/WEB/0]OUT 2016-11-04 14:55:44.188 DEBUG 10 --- [tcp-client-io-2] cloudfoundry-client.token                : JWT Token: xxxxx2
    
    # access token expiration, get a new one using refresh token -> FAILED (refresh token has expired too)
    2016-11-04T16:05:33.25+0100 [APP/PROC/WEB/0]OUT 2016-11-04 15:05:33.251 DEBUG 10 --- [tcp-client-io-1] cloudfoundry-client.request              : GET    https://login.YYYYYYYY:443/oauth/token?grant_type=refresh_token&client_id=xxxx&client_secret=&refresh_token=xxx
    2016-11-04T16:05:33.25+0100 [APP/PROC/WEB/0]OUT 2016-11-04 15:05:33.252 DEBUG 10 --- [tcp-client-io-1] cloudfoundry-client.response             : 401    https://login.YYYYYYYY:443/oauth/token?grant_type=refresh_token&client_id=xxxxx&client_secret=&refresh_token=xxx
    
    

    Notice, that refresh token expiration return HTTP 401 error but no exception seems to be raised.

    Latter, when performing action using cf java client, got the following exception :

    2016-11-04T16:07:46.06+0100 [APP/PROC/WEB/0]OUT 2016-11-04 15:07:46.062 DEBUG 10 --- [tcp-client-io-2] cloudfoundry-client.request              : GET    https://api.xxxxxxxx:443/v2/apps/zzzz
    2016-11-04T16:07:46.06+0100 [APP/PROC/WEB/0]OUT 2016-11-04 15:07:46.063 DEBUG 10 --- [tcp-client-io-2] cloudfoundry-client.response             : 401    https://api.xxxxxxx:443/v2/apps/zzzz
    2016-11-04T16:07:46.07+0100 [APP/PROC/WEB/0]OUT 2016-11-04 15:07:46.072 ERROR 10 --- [tcp-client-io-2] c.o.c.s.f.s.filter.CreateSecurityGroup   : Fail to create security group. Error details {}
    2016-11-04T16:07:46.07+0100 [APP/PROC/WEB/0]OUT org.cloudfoundry.client.v2.CloudFoundryException: CF-InvalidAuthToken(1000): Invalid Auth Token
    2016-11-04T16:07:46.07+0100 [APP/PROC/WEB/0]OUT         at org.cloudfoundry.reactor.client.CloudFoundryExceptionBuilder.lambda$build$0(CloudFoundryExceptionBuilder.java:52) ~[cloudfoundry-client-reactor-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
    2016-11-04T16:07:46.07+0100 [APP/PROC/WEB/0]OUT         at reactor.core.publisher.MonoThenMap$MonoThenApplyMain.onNext(MonoThenMap.java:98) ~[reactor-core-3.0.3.RELEASE.jar!/:na]
    2016-11-04T16:07:46.07+0100 [APP/PROC/WEB/0]OUT         at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) ~[reactor-core-3.0.3.RELEASE.jar!/:na]
    
  12. added a commit that references this issue on Nov 10, 2016
  13. added a commit that references this issue on Jan 18, 2017
  14. nebhale commented on Jan 18, 2017

    @nebhale
    Contributor

    @mohanpindyala @s-bortolussi The latest snapshot should solve all problems around token negotiation and refresh. If you could give the snapshot a try and report back with any issues, it'd be greatly appreciated.

  15. 5 remaining items

  16. nebhale commented on Jan 21, 2017

    @nebhale
    Contributor

    @ahgittin The negotiation has been updated so that a 401 triggers it every time. So, you'd expect to see a failed call, then a negotiation of a new token, and then retrying the original call. There's some fallback logic if the refresh token has expired, so possibly even two failed called before a successful retry would be expected. I'd be interested to see what's causing your hard failures though. If you set the cloudfoundry-client logger to DEBUG while it's running, it'll give detailed information about what calls are being made and what the responses are. This should be enough for me to get an idea of whether this is something systemic that we can solve.

  17. ahgittin commented on Jan 23, 2017

    @ahgittin

    @nebhale Here are logs from the REST server.

    2017-01-22 02:07:29,866 [reactor-http-nio-1] DEBUG cloudfoundry-client.response - 200    https://api.local.pcfdev.io:443/v2/service_instances?q=service_plan_guid%20IN%209d58d708-e170-469f-b3ea-6663a9c6a532,2b05a2fd-90ad-4673-8211-a6a9ba6df96f
    

    Things were working fine. Suspended laptop. Resumed 1.5h later and refreshed browser page.

    2017-01-22 03:36:30,938 [qtp469643779-21] DEBUG cloudfoundry-client.request - GET    https://api.local.pcfdev.io:443/v2/organizations
    2017-01-22 03:36:30,941 [qtp469643779-19] DEBUG cloudfoundry-client.request - GET    https://api.local.pcfdev.io:443/v2/service_plan_visibilities
    2017-01-22 03:36:30,942 [qtp469643779-20] DEBUG cloudfoundry-client.request - GET    https://api.local.pcfdev.io:443/v2/service_brokers
    2017-01-22 03:36:31,672 [qtp469643779-21] ERROR i.c.s.error.ExceptionController - error: {}
    reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
    2017-01-22 03:36:31,672 [qtp469643779-19] ERROR i.c.s.error.ExceptionController - error: {}
    reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
    2017-01-22 03:38:30,785 [qtp469643779-21] DEBUG cloudfoundry-client.request - GET    https://api.local.pcfdev.io:443/v2/service_brokers
    2017-01-22 03:38:31,710 [qtp469643779-47] DEBUG cloudfoundry-client.request - GET    https://api.local.pcfdev.io:443/v2/service_brokers
    

    Web client (where server making REST calls) stuck in loading state. Suspended laptop after a little bit. Resumed this morning, refreshed browser, and things seemed healthy.

    2017-01-23 10:14:46,393 [qtp469643779-191] DEBUG cloudfoundry-client.request - GET    https://api.local.pcfdev.io:443/v2/organizations
    2017-01-23 10:14:46,393 [qtp469643779-190] DEBUG cloudfoundry-client.request - GET    https://api.local.pcfdev.io:443/v2/service_brokers
    2017-01-23 10:14:46,393 [qtp469643779-192] DEBUG cloudfoundry-client.request - GET    https://api.local.pcfdev.io:443/v2/service_plan_visibilities
    2017-01-23 10:14:46,714 [reactor-http-nio-1] DEBUG cloudfoundry-client.response - 401    https://api.local.pcfdev.io:443/v2/organizations
    2017-01-23 10:14:46,717 [reactor-http-nio-1] DEBUG cloudfoundry-client.request - GET    https://api.local.pcfdev.io:443/v2/organizations
    2017-01-23 10:14:46,721 [reactor-http-nio-1] DEBUG cloudfoundry-client.response - 401    https://api.local.pcfdev.io:443/v2/service_brokers
    2017-01-23 10:14:46,721 [reactor-http-nio-1] DEBUG cloudfoundry-client.request - GET    https://api.local.pcfdev.io:443/v2/service_brokers
    2017-01-23 10:14:46,722 [reactor-http-nio-1] DEBUG cloudfoundry-client.response - 401    https://api.local.pcfdev.io:443/v2/service_plan_visibilities
    2017-01-23 10:14:46,722 [reactor-http-nio-1] DEBUG cloudfoundry-client.request - GET    https://api.local.pcfdev.io:443/v2/service_plan_visibilities
    2017-01-23 10:14:46,760 [reactor-http-nio-1] DEBUG cloudfoundry-client.token - Negotiating using refresh token
    2017-01-23 10:14:46,763 [reactor-http-nio-1] DEBUG cloudfoundry-client.request - POST   https://uaa.local.pcfdev.io:443/oauth/token
    2017-01-23 10:14:47,005 [reactor-http-nio-1] DEBUG cloudfoundry-client.response - 200    https://uaa.local.pcfdev.io:443/oauth/token
    2017-01-23 10:14:47,019 [reactor-http-nio-1] DEBUG cloudfoundry-client.token - Refresh Token: fb25787ff48946839ef23656bb119eed-r
    2017-01-23 10:14:47,019 [reactor-http-nio-1] DEBUG cloudfoundry-client.token - Access Token:  bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImtleS0xIiwidHlwIjoiSldUIn0.eyJqdGkiOiJkNzRkOGViOWFlMWE0NzM3YmMzN2Q3OGZkMjYzYTk2OSIsInN1YiI6ImUwYWNkZDFmLWMwMmQtNDBlMy1iNTZlLTQ1MWQxMWRkODEzNCIsInNjb3BlIjpbInJvdXRpbmcucm91dGVyX2dyb3Vwcy5yZWFkIiwiY2xvdWRfY29udHJvbGxlci5yZWFkIiwicGFzc3dvcmQud3JpdGUiLCJjbG91ZF9jb250cm9sbGVyLndyaXRlIiwib3BlbmlkIiwicm91dGluZy5yb3V0ZXJfZ3JvdXBzLndyaXRlIiwiZG9wcGxlci5maXJlaG9zZSIsInNjaW0ud3JpdGUiLCJzY2ltLnJlYWQiLCJjbG91ZF9jb250cm9sbGVyLmFkbWluIiwidWFhLnVzZXIiXSwiY2xpZW50X2lkIjoiY2YiLCJjaWQiOiJjZiIsImF6cCI6ImNmIiwiZ3JhbnRfdHlwZSI6InBhc3N3b3JkIiwidXNlcl9pZCI6ImUwYWNkZDFmLWMwMmQtNDBlMy1iNTZlLTQ1MWQxMWRkODEzNCIsIm9yaWdpbiI6InVhYSIsInVzZXJfbmFtZSI6ImFkbWluIiwiZW1haWwiOiJhZG1pbiIsInJldl9zaWciOiI5MTMzZjMzIiwiaWF0IjoxNDg0OTUyNzM3LCJleHAiOjE0ODQ5NTMzMzcsImlzcyI6Imh0dHBzOi8vdWFhLmxvY2FsLnBjZmRldi5pby9vYXV0aC90b2tlbiIsInppZCI6InVhYSIsImF1ZCI6WyJzY2ltIiwiY2xvdWRfY29udHJvbGxlciIsInBhc3N3b3JkIiwiY2YiLCJ1YWEiLCJvcGVuaWQiLCJkb3BwbGVyIiwicm91dGluZy5yb3V0ZXJfZ3JvdXBzIl19.C9NuyFKC8AWXKx_DTK5rszIiT8dJO9qCJVMfI0W4SJjjbpt8YdpNQJl7k_EtnucnhukqmghoeUGrLhKKYFtVSFNnHw3XNLeBRAqeF4elAp8kFEosZ7E8_h9ZMAoa_NBNiU8wvqESS2g2nM9ub8WiPFV4MaHpNEEsa3e64FgsWpw
    2017-01-23 10:14:47,040 [reactor-http-nio-1] DEBUG cloudfoundry-client.response - 200    https://api.local.pcfdev.io:443/v2/service_brokers
    

    They don't seem to say very much. Stopped the server and got these stack traces:

    2017-01-23 10:18:58,343 [qtp469643779-47] ERROR i.c.s.error.ExceptionController - error: {}
    reactor.core.Exceptions$ReactiveException: java.lang.InterruptedException
    	at reactor.core.Exceptions.propagate(Exceptions.java:218)
    	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:84)
    	at reactor.core.publisher.Mono.block(Mono.java:1173)
    	at io.cloudsoft.servicebroker.service.CloudFoundryController.serviceBrokers(CloudFoundryController.java:85)
    	at ... (suppressed spring/mvc trace)
    Caused by: java.lang.InterruptedException: null
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
    	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
    	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:81)
    	... 100 common frames omitted
    

    and

    2017-01-23 10:18:58,343 [qtp469643779-20] ERROR i.c.s.error.ExceptionController - error: {}
    reactor.core.Exceptions$ReactiveException: java.lang.InterruptedException
    	at reactor.core.Exceptions.propagate(Exceptions.java:218)
    	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:84)
    	at reactor.core.publisher.Mono.block(Mono.java:1173)
    	at io.cloudsoft.servicebroker.service.CloudFoundryController.serviceBrokers(CloudFoundryController.java:85)
    	at ... (suppressed)
    Caused by: java.lang.InterruptedException: null
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
    	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
    	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:81)
    	... 100 common frames omitted
    

    and

    2017-01-23 10:18:58,343 [qtp469643779-21] ERROR i.c.s.error.ExceptionController - error: {}
    reactor.core.Exceptions$ReactiveException: java.lang.InterruptedException
    	at reactor.core.Exceptions.propagate(Exceptions.java:218)
    	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:84)
    	at reactor.core.publisher.Mono.block(Mono.java:1173)
    	at io.cloudsoft.servicebroker.service.CloudFoundryController.serviceBrokers(CloudFoundryController.java:85)
    	at ... (suppressed)
    Caused by: java.lang.InterruptedException: null
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
    	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
    	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:81)
    	... 100 common frames omitted
    

    Looks like there was some sort of lock among threads 47, 20, and 21 when the problem occurred. (Not sure what happened to 19.) So guessing the logic for refresh isn't 100% robust when the client is used concurrently in multiple threads?

  18. ahgittin commented on Jan 23, 2017

    @ahgittin

    (Long message with lots of evidence. Jump to the end for the TL;DR.)

    With better logging and further inspection, when using our web app calling to our REST server which uses cf-java-client (current 2.3.0 snapshot), after a delay of 1h or more, we sometimes get the following:

    The Connection has been aborted by the remote peer reported by the cf-java-client is not caught there, and not caught in our REST code, and returned to the browser as a 500.

    {"timestamp":1485172964674,"status":500,"error":"Internal Server Error","exception":"reactor.ipc.netty.channel.ContextHandler$1","message":"Connection has been aborted by the remote peer","path":"/cf/organizations"}

    This is in the following method:

        @RequestMapping("/organizations")
        public List<OrganizationResource> organizations() {
            LOG.debug("/organizations, calling CF client to list");
            try {
                return cloudFoundryClient.organizations()
                    .list(ListOrganizationsRequest.builder().build())
                    .map(listOrganizationsResponse -> listOrganizationsResponse.getResources())
                    .block();
            } finally {
                LOG.debug("/organizations, finished call to CF client to list");
            }
        }
    

    We observe the following in the log:

    2017-01-23 12:02:44,198 DEBUG [qtp1774795940-21] i.c.s.service.CloudFoundryController  /service_brokers, calling CF client to list
    2017-01-23 12:02:44,198 DEBUG [qtp1774795940-45] i.c.s.service.CloudFoundryController  /service_plan_visibilities, calling CF client to list
    2017-01-23 12:02:44,198 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController  /organizations, calling CF client to list
    2017-01-23 12:02:44,240 DEBUG [qtp1774795940-21] cloudfoundry-client.request  GET    https://api.local.pcfdev.io:443/v2/service_brokers
    2017-01-23 12:02:44,240 DEBUG [qtp1774795940-20] cloudfoundry-client.request  GET    https://api.local.pcfdev.io:443/v2/organizations
    2017-01-23 12:02:44,240 DEBUG [qtp1774795940-45] cloudfoundry-client.request  GET    https://api.local.pcfdev.io:443/v2/service_plan_visibilities
    2017-01-23 12:02:44,487 DEBUG [qtp1774795940-21] i.c.s.service.CloudFoundryController  /service_brokers, finished call to CF client to list
    2017-01-23 12:02:44,488 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController  /organizations, finished call to CF client to list
    2017-01-23 12:02:44,490 DEBUG [qtp1774795940-45] i.c.s.service.CloudFoundryController  /service_plan_visibilities, finished call to CF client to list
    2017-01-23 12:02:44,527 ERROR [qtp1774795940-45] i.c.s.error.ExceptionController  error: {}
    reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
    2017-01-23 12:02:44,527 ERROR [qtp1774795940-20] i.c.s.error.ExceptionController  error: {}
    reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
    2017-01-23 12:02:44,527 ERROR [qtp1774795940-21] i.c.s.error.ExceptionController  error: {}
    reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
    

    Waiting a few minutes and reloading our web app triggers the following in the log (with no intervening messages):

    2017-01-23 12:10:09,125 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController  /service_brokers, calling CF client to list
    2017-01-23 12:10:09,136 DEBUG [qtp1774795940-47] cloudfoundry-client.request  GET    https://api.local.pcfdev.io:443/v2/service_brokers
    2017-01-23 12:10:09,164 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController  /service_brokers, finished call to CF client to list
    2017-01-23 12:10:09,166 ERROR [qtp1774795940-47] i.c.s.error.ExceptionController  error: {}
    reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
    2017-01-23 12:10:09,196 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController  /organizations, calling CF client to list
    2017-01-23 12:10:09,198 DEBUG [qtp1774795940-20] cloudfoundry-client.request  GET    https://api.local.pcfdev.io:443/v2/organizations
    2017-01-23 12:10:09,199 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController  /service_plan_visibilities, calling CF client to list
    2017-01-23 12:10:09,200 DEBUG [qtp1774795940-47] cloudfoundry-client.request  GET    https://api.local.pcfdev.io:443/v2/service_plan_visibilities
    2017-01-23 12:10:09,206 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController  /organizations, finished call to CF client to list
    2017-01-23 12:10:09,212 ERROR [qtp1774795940-20] i.c.s.error.ExceptionController  error: {}
    reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
    2017-01-23 12:10:09,302 DEBUG [reactor-http-nio-1] cloudfoundry-client.response  401    https://api.local.pcfdev.io:443/v2/service_plan_visibilities
    2017-01-23 12:10:09,307 DEBUG [reactor-http-nio-1] cloudfoundry-client.request  GET    https://api.local.pcfdev.io:443/v2/service_plan_visibilities
    2017-01-23 12:10:09,360 DEBUG [reactor-http-nio-1] cloudfoundry-client.token  Negotiating using refresh token
    2017-01-23 12:10:09,367 DEBUG [reactor-http-nio-1] cloudfoundry-client.request  POST   https://uaa.local.pcfdev.io:443/oauth/token
    2017-01-23 12:10:09,593 DEBUG [reactor-http-nio-1] cloudfoundry-client.response  200    https://uaa.local.pcfdev.io:443/oauth/token
    2017-01-23 12:10:09,602 DEBUG [reactor-http-nio-1] cloudfoundry-client.token  Refresh Token: c89c9fafe5f64177a3afd959ecf90e16-r
    2017-01-23 12:10:09,602 DEBUG [reactor-http-nio-1] cloudfoundry-client.token  Access Token:  bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImtleS0xIiwidHlwIjoiSldUIn0.eyJqdGkiOiJiNTMwMGJkY2FkNWU0Mjg0YTBiZWRmNTQ3ZjdlYmVmMiIsInN1YiI6ImUwYWNkZDFmLWMwMmQtNDBlMy1iNTZlLTQ1MWQxMWRkODEzNCIsInNjb3BlIjpbInJvdXRpbmcucm91dGVyX2dyb3Vwcy5yZWFkIiwiY2xvdWRfY29udHJvbGxlci5yZWFkIiwicGFzc3dvcmQud3JpdGUiLCJjbG91ZF9jb250cm9sbGVyLndyaXRlIiwib3BlbmlkIiwicm91dGluZy5yb3V0ZXJfZ3JvdXBzLndyaXRlIiwiZG9wcGxlci5maXJlaG9zZSIsInNjaW0ud3JpdGUiLCJzY2ltLnJlYWQiLCJjbG91ZF9jb250cm9sbGVyLmFkbWluIiwidWFhLnVzZXIiXSwiY2xpZW50X2lkIjoiY2YiLCJjaWQiOiJjZiIsImF6cCI6ImNmIiwiZ3JhbnRfdHlwZSI6InBhc3N3b3JkIiwidXNlcl9pZCI6ImUwYWNkZDFmLWMwMmQtNDBlMy1iNTZlLTQ1MWQxMWRkODEzNCIsIm9yaWdpbiI6InVhYSIsInVzZXJfbmFtZSI6ImFkbWluIiwiZW1haWwiOiJhZG1pbiIsInJldl9zaWciOiI5MTMzZjMzIiwiaWF0IjoxNDg0OTU5NjYwLCJleHAiOjE0ODQ5NjAyNjAsImlzcyI6Imh0dHBzOi8vdWFhLmxvY2FsLnBjZmRldi5pby9vYXV0aC90b2tlbiIsInppZCI6InVhYSIsImF1ZCI6WyJzY2ltIiwiY2xvdWRfY29udHJvbGxlciIsInBhc3N3b3JkIiwiY2YiLCJ1YWEiLCJvcGVuaWQiLCJkb3BwbGVyIiwicm91dGluZy5yb3V0ZXJfZ3JvdXBzIl19.tLRauKyT5wXDlnqbuvE84luWxNCASbVT9Td6DYKg15ocjdxDzl4_vjkJenlkZQ4-DwNZOy08sC4MQGA1sQ8EMYFS_6v8CYjJS3ao9rurR5Qn7wIvZKh0fyI46nEXcUZ8EUMUHFenRkNpaSmGS6B2r3IeGQPSHSIGsnltsH6CjJI
    2017-01-23 12:10:09,615 DEBUG [reactor-http-nio-1] cloudfoundry-client.response  200    https://api.local.pcfdev.io:443/v2/service_plan_visibilities
    2017-01-23 12:10:09,630 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController  /service_plan_visibilities, finished call to CF client to list
    

    This time one of the three calls to CF gets the 401 and triggers the token refresh. The webapp is now healthy (after a reload).

    So it seems the problem is not any concurrency but rather that the connection to the CF server is sometimes aborted, allegedly by the remote peer (the CF server). In those cases no 401 is returned, the token not refreshed, and calls to/by the cf-java-client will continue to fail until it gets the expected 401.

    There is no apparent pattern in CF returning a 401 vs a connection aborted. I am using PCF Dev on my laptop so vbox or pcf-dev could be implicated (although I'm always waiting a bit of time after resuming).

    tl;dr

    Things right themselves so this isn't severe. If the connection aborted is a known issue with PCF Dev only, then we could park this as a known issue with dev environments. OTOH if that behaviour happens in real CF systems then this probably should be fixed, either here or in CF, depending under what circumstances it is allowed to abort a connection.

  19. twoseat commented on Jan 23, 2017

    @twoseat
    Contributor

    @ahgittin I believe part of what you're seeing is due to pcfdev not handling sleeping - we have several tests that will fail with a similar time-related error if pcfdev has been slept. Naturally that's not something we'd expect to see in production (!) but it may be triggering a legitimate problem. I'll let @nebhale comment further...

  20. ahgittin commented on Jan 23, 2017

    @ahgittin

    A bit more info. If it's a pcf-dev sleep issue, it isn't a straightforward one.

    Making one more call from the browser after the above triggers this:

    2017-01-23 12:10:09,125 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController  /service_brokers, calling CF client to list
    2017-01-23 12:10:09,136 DEBUG [qtp1774795940-47] cloudfoundry-client.request  GET    https://api.local.pcfdev.io:443/v2/service_brokers
    2017-01-23 12:10:09,164 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController  /service_brokers, finished call to CF client to list
    2017-01-23 12:10:09,166 ERROR [qtp1774795940-47] i.c.s.error.ExceptionController  error: {}
    reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
    2017-01-23 12:10:09,196 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController  /organizations, calling CF client to list
    2017-01-23 12:10:09,198 DEBUG [qtp1774795940-20] cloudfoundry-client.request  GET    https://api.local.pcfdev.io:443/v2/organizations
    2017-01-23 12:10:09,199 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController  /service_plan_visibilities, calling CF client to list
    

    And then coming back to it 30m later with no intervening sleep and doing a refresh gives:

    2017-01-23 12017-01-23 12:40:30,992 DEBUG [qtp1774795940-21] i.c.s.service.CloudFoundryController  /service_brokers, calling CF client to list
    2017-01-23 12:40:31,005 DEBUG [qtp1774795940-21] cloudfoundry-client.request  GET    https://api.local.pcfdev.io:443/v2/service_brokers
    2017-01-23 12:40:31,031 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController  /organizations, calling CF client to list
    2017-01-23 12:40:31,033 DEBUG [qtp1774795940-20] cloudfoundry-client.request  GET    https://api.local.pcfdev.io:443/v2/organizations
    2017-01-23 12:40:31,044 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController  /service_plan_visibilities, calling CF client to list
    2017-01-23 12:40:31,045 DEBUG [qtp1774795940-47] cloudfoundry-client.request  GET    https://api.local.pcfdev.io:443/v2/service_plan_visibilities
    2017-01-23 12:40:31,163 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController  /service_plan_visibilities, finished call to CF client to list
    2017-01-23 12:40:31,165 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController  /organizations, finished call to CF client to list
    2017-01-23 12:40:31,166 ERROR [qtp1774795940-47] i.c.s.error.ExceptionController  error: {}
    reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
    2017-01-23 12:40:31,169 ERROR [qtp1774795940-20] i.c.s.error.ExceptionController  error: {}
    reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
    2017-01-23 12:40:31,194 DEBUG [reactor-http-nio-1] cloudfoundry-client.response  401    https://api.local.pcfdev.io:443/v2/service_brokers
    2017-01-23 12:40:31,194 DEBUG [reactor-http-nio-1] cloudfoundry-client.request  GET    https://api.local.pcfdev.io:443/v2/service_brokers
    2017-01-23 12:40:31,212 DEBUG [reactor-http-nio-1] cloudfoundry-client.token  Negotiating using refresh token
    2017-01-23 12:40:31,213 DEBUG [reactor-http-nio-1] cloudfoundry-client.request  POST   https://uaa.local.pcfdev.io:443/oauth/token
    2017-01-23 12:40:31,458 DEBUG [reactor-http-nio-1] cloudfoundry-client.response  200    https://uaa.local.pcfdev.io:443/oauth/token
    2017-01-23 12:40:31,460 DEBUG [reactor-http-nio-1] cloudfoundry-client.token  Refresh Token: c89c9fafe5f64177a3afd959ecf90e16-r
    2017-01-23 12:40:31,460 DEBUG [reactor-http-nio-1] cloudfoundry-client.token  Access Token:  bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImtleS0xIiwidHlwIjoiSldUIn0.eyJqdGkiOiI4YjZlZTU2MWIyZmI0ZTg0ODFkODllNjU5MjNhNjhiNyIsInN1YiI6ImUwYWNkZDFmLWMwMmQtNDBlMy1iNTZlLTQ1MWQxMWRkODEzNCIsInNjb3BlIjpbInJvdXRpbmcucm91dGVyX2dyb3Vwcy5yZWFkIiwiY2xvdWRfY29udHJvbGxlci5yZWFkIiwicGFzc3dvcmQud3JpdGUiLCJjbG91ZF9jb250cm9sbGVyLndyaXRlIiwib3BlbmlkIiwicm91dGluZy5yb3V0ZXJfZ3JvdXBzLndyaXRlIiwiZG9wcGxlci5maXJlaG9zZSIsInNjaW0ud3JpdGUiLCJzY2ltLnJlYWQiLCJjbG91ZF9jb250cm9sbGVyLmFkbWluIiwidWFhLnVzZXIiXSwiY2xpZW50X2lkIjoiY2YiLCJjaWQiOiJjZiIsImF6cCI6ImNmIiwiZ3JhbnRfdHlwZSI6InBhc3N3b3JkIiwidXNlcl9pZCI6ImUwYWNkZDFmLWMwMmQtNDBlMy1iNTZlLTQ1MWQxMWRkODEzNCIsIm9yaWdpbiI6InVhYSIsInVzZXJfbmFtZSI6ImFkbWluIiwiZW1haWwiOiJhZG1pbiIsInJldl9zaWciOiI5MTMzZjMzIiwiaWF0IjoxNDg0OTYxNDgxLCJleHAiOjE0ODQ5NjIwODEsImlzcyI6Imh0dHBzOi8vdWFhLmxvY2FsLnBjZmRldi5pby9vYXV0aC90b2tlbiIsInppZCI6InVhYSIsImF1ZCI6WyJzY2ltIiwiY2xvdWRfY29udHJvbGxlciIsInBhc3N3b3JkIiwiY2YiLCJ1YWEiLCJvcGVuaWQiLCJkb3BwbGVyIiwicm91dGluZy5yb3V0ZXJfZ3JvdXBzIl19.A5R5S4tC_s3q4kwaFp18cxnex1wM5ul87KYDD-qsvZ3ie5QW_EOeVfOAGupt3ncZzPXO7bwCvez36-U1-UxM8gMn_1ZOu7PHMjVp-gSycQfTxfzeyhecDAXl3Tn0MKAO-78KwzoR4YLNcgCaIGd-2Lia3-O-If7oC72ZbEGAxxw
    2017-01-23 12:40:31,473 DEBUG [reactor-http-nio-1] cloudfoundry-client.response  200    https://api.local.pcfdev.io:443/v2/service_brokers
    2017-01-23 12:40:31,487 DEBUG [qtp1774795940-21] i.c.s.service.CloudFoundryController  /service_brokers, finished call to CF client to list
    

    The webapp is unhappy due to the errors, but fine on the next refresh:

    2017-01-23 12:40:42,988 DEBUG [qtp1774795940-50] i.c.s.service.CloudFoundryController  /service_brokers, calling CF client to list
    2017-01-23 12:40:42,989 DEBUG [qtp1774795940-50] cloudfoundry-client.request  GET    https://api.local.pcfdev.io:443/v2/service_brokers
    2017-01-23 12:40:43,014 DEBUG [reactor-http-nio-1] cloudfoundry-client.response  200    https://api.local.pcfdev.io:443/v2/service_brokers
    2017-01-23 12:40:43,015 DEBUG [qtp1774795940-50] i.c.s.service.CloudFoundryController  /service_brokers, finished call to CF client to list
    2017-01-23 12:40:43,076 DEBUG [qtp1774795940-21] i.c.s.service.CloudFoundryController  /organizations, calling CF client to list
    2017-01-23 12:40:43,078 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController  /service_plan_visibilities, calling CF client to list
    2017-01-23 12:40:43,079 DEBUG [qtp1774795940-20] cloudfoundry-client.request  GET    https://api.local.pcfdev.io:443/v2/service_plan_visibilities
    2017-01-23 12:40:43,080 DEBUG [qtp1774795940-21] cloudfoundry-client.request  GET    https://api.local.pcfdev.io:443/v2/organizations
    2017-01-23 12:40:43,096 DEBUG [reactor-http-nio-1] cloudfoundry-client.response  200    https://api.local.pcfdev.io:443/v2/service_plan_visibilities
    2017-01-23 12:40:43,097 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController  /service_plan_visibilities, finished call to CF client to list
    2017-01-23 12:40:43,102 DEBUG [reactor-http-nio-1] cloudfoundry-client.response  200    https://api.local.pcfdev.io:443/v2/organizations
    2017-01-23 12:40:43,115 DEBUG [qtp1774795940-21] i.c.s.service.CloudFoundryController  /organizations, finished call to CF client to list
    

    (BTW all my log messages since 2017-01-23 12:02 are included; I've just broken them in to chunks for context.)

    Note that:

    • Our server has a single CF client and token provider bean, so I wouldn't expect the call at
      2017-01-23 12:10:09,136 to fail just after token refresh.
    • The system has been up for 30 minutes at 12:40 so I'd be hesitant to assume that the connection aborted happening then is due to the vbox sleep, particularly as just a couple seconds later once the token is established, the same request works fine. (Also the cf CLI works fine for the same calls.)

    HTH

  21. nebhale commented on Jan 23, 2017

    @nebhale
    Contributor

    @ahgittin Thanks for all the research; it's quite helpful. I believe that there are a couple of things going on here. In general, we'd expect to see 401s on the first request as well as when the current token has expired. It's possible to get a multiple 401s simultaneously that all resolve new tokens if requests are made concurrently (see the triple that you observe quite a lot). There's no reasonable solution to this repetitive resolution without synchronizing access to the token, so I think we're making the correct choice there.

    Secondly, there is always the case that the server will terminate connections for unknown reasons. It could come from the server being over loaded (the Cloud Controller isn't the most scalable thing in the world), network devices (such as the AWS ELB) terminating connections, and even, sleeping a machine with an open connection so that the server terminates the connection, but the client doesn't know that when it starts up. In general, you probably need to handle these intermittent network failures directly as part of your code simply to be a good citizen in a massively network world; i.e. you should expect failure and account for it.

    That being said, we're seeing this kind of failure a bit more than we used to and it coincides with recent changes to reactor-netty. @smaldini can you please consider this independent verification of the issues that @markpollack has reported. This is a lot of experimentation and detailed timings that may help us get to the bottom of what's going on. We'll also start taking a look at re-requesting "Connection aborted errors" for a short time by default in our code to take some of the load off users.

  22. ahgittin commented on Jan 23, 2017

    @ahgittin

    @nebhale @twoseat Sounds reasonable. Only thing I disagree with -- and weakly -- is whether Java consumers of the client, ie ourselves, should be doing a retry, if you were suggesting that. We give failure messages on failure. Feels like a violation of the contract if our Java code starts retrying, either we risk doing it pointlessly a lot of the time, or we put in nasty things in like if ((error instanceof reactor.ipc.netty.channel.ContextHandler$1) && error.getMessage().contains("Connection has been aborted by the remote peer")). However I totally accept JS client could retry with increasing delays, if we're really concerned about the user refreshing. :)

    You've definitely fixed the major issue. I'd say close this, and maybe link to the issues you mention from @markpollack. What remains is simply a nice-to-have. Good luck resolving it and ping me if there's another build you want us to look at.

  23. nebhale commented on Jan 23, 2017

    @nebhale
    Contributor

    @ahgittin I absolutely am suggesting that. One of the main tenants of microservices is that you accept that failure happens and you have a strategy for dealing with it. This is most obvious example of this is the entire Circuit Breaker Pattern popularized by Netflix Hystrix. The main reason I've resisted retrying automatically so far, is that everyone has an idea of how long, how many times, what delay algorithm, etc. to use. When we talk about retry we're really hoping for something like:

    .retryWhen(DelayUtils.exponentialBackOffError(Duration.ofMillis(100), Duration.ofMillis(250), Duration.ofSeconds(3)))
    

    Agreed that this would need to be combined with some brutal if/then to determine whether to apply that, but we can either add another version of DelayUtils.exponentialBackOffError that matches on exception types, or we can ask for a similar operator in Reactor to otherwise(Class<E>, Function<? super E, ? extends Mono<? extends T>>). At the very least, I'll ask @smaldini to give us a real exception so that it's reliably identifiable ($1 could change tomorrow just be moving code around).

  24. ahgittin commented on Jan 23, 2017

    @ahgittin

    Update: the call can hang; I suggest confirming that is also a known symptom of the reactor-netty issues -- and not caused by the token refresh -- before closing this issue

    More detail... it seems there are at least four relevant things that we've observed happen when we call cloudFoundryClient.organizations()....block() (snippet above):

    • It succeeds
    • It returns 401 (confirmed fixed -- this is now intercepted and token refreshed and resubmitted, so we no longer see this!)
    • Throws "connection aborted" (due to reactor-netty ?)
    • Hangs

    The "hang" is new, in this comment, so I'm including details. Observed with same 2.3.0.BUILD-SNAPSHOT, less often than the other behaviours but a bit more severe when it happens.

    The log shows:

    2017-01-23 17:02:50,981 DEBUG [qtp1675239551-48] i.c.s.service.CloudFoundryController  /organizations, calling CF client to list
    2017-01-23 17:02:50,982 DEBUG [qtp1675239551-48] cloudfoundry-client.request  GET    https://api.local.pcfdev.io:443/v2/organizations
    

    The corresponding completion messages are never displayed, implying that the thread is hung.

    jstack shows this:

    "qtp1675239551-48" #48 prio=5 os_prio=31 tid=0x00007fd543075800 nid=0x8f03 waiting on condition [0x0000700011375000]
       java.lang.Thread.State: WAITING (parking)
            at sun.misc.Unsafe.park(Native Method)
            - parking to wait for  <0x0000000778cc7040> (a java.util.concurrent.CountDownLatch$Sync)
            at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
            at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
            at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:81)
            at reactor.core.publisher.Mono.block(Mono.java:1173)
            at io.cloudsoft.servicebroker.service.CloudFoundryController.organizations(CloudFoundryController.java:159)
           (removed the rest, spring MVC)
    

    I've made multiple curl calls to trigger this code, and each generates a new such hung thread as above.

    Making other calls seems to refresh things sufficiently that this call subsequently works, so it is recoverable, although there will be a thread leak as the hung threads such as the above do not seem to be resolved after success. (Maybe they will be on session expiry / logout, if you track calls and run some scavenger.)

  25. ahgittin commented on Jan 23, 2017

    @ahgittin

    @nebhale Thanks for clarifying. In our case the java is a fairly minimal proxy. Our JS client needs to be able to handle failures no matter how much smarts we put in our proxy, ideally with the exponential back-off retry you suggest (though currently it just reports failure to the user who can retry, which is fine for now) so marginal benefit putting it also in the java. If our java tracked state I'd agree with you.

    It would be nice if cf-java-client provides a retry mechanism (with fairly low timeout bounds and/or configurable, though in both cases it seems a clunky contract) or richer failure semantics (definitely no $1!). But clearly the best fix is to sort out whatever is going on between reactor-netty and CF server -- especially as it is sometimes leaking threads as above, and it looks like the cf CLI is not encountering these issues. (Implementing retry is just going to compound the problem here! :P )

  26. ahgittin commented on Jan 27, 2017

    @ahgittin

    @nebhale now after delay JWT expiration is the cause of an aborted connection, client side, with the latest current snapshot of the client (PCF Dev has been suspended and resumed) -- not sure if this was contributing to the previous aborts or not. it is new today and the only thing changed is the cf-java-client version (LMK if this should be separate issue). HTH.

    2017-01-27 17:08:35,550 ERROR [HttpClient] Error processing connection. Requesting close the channel
    io.jsonwebtoken.ExpiredJwtException: JWT expired at 2017-01-27T15:16:18Z. Current time: 2017-01-27T17:08:35Z, a difference of 6737534 milliseconds.  Allowed clock skew: 0 milliseconds.
    	at io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:385)
    	at io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:481)
    	at io.jsonwebtoken.impl.DefaultJwtParser.parseClaimsJwt(DefaultJwtParser.java:514)
    	at org.cloudfoundry.reactor.tokenprovider.AbstractUaaTokenProvider.parseToken(AbstractUaaTokenProvider.java:147)
    	at org.cloudfoundry.reactor.tokenprovider.AbstractUaaTokenProvider.extractAccessToken(AbstractUaaTokenProvider.java:128)
    
  27. nebhale commented on Jan 27, 2017

    @nebhale
    Contributor

    @ahgittin That's the addition of some client-side logging that should only happen when the token is first issued. Is it possible that the clock inside the pcfdev instances has skewed? Either way, I'll protect against this exception to ensure that the debug logging doesn't kill anything.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions