You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
reactor.core.Exceptions$BubblingException: org.cloudfoundry.client.v2.CloudFoundryException: CF-InvalidAuthToken(1000): Invalid Auth Token
at reactor.core.Exceptions.bubble(Exceptions.java:96)
at reactor.core.publisher.Operators.onErrorDropped(Operators.java:317)
at reactor.core.publisher.LambdaSubscriber.onError(LambdaSubscriber.java:126)
at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:193)
at reactor.core.publisher.MonoFlatMap$FlattenSubscriber.onError(MonoFlatMap.java:166)
at reactor.core.publisher.MonoCollectList$MonoBufferAllSubscriber.onError(MonoCollectList.java:100)
at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:134)
Can you please turn the logger cloudfoundry-client to DEBUG. This should start generating a lot of information about the requests made to the server including negotiation of the token. It'd be helpful to get the contents of these outputs (you can filter out everything else that's application specific) from the start of the application (and the first negotiation of a token) all the way to the moment that the token is not longer accepted.
In addition to that logging, please update to the latest version (2.0.1.RELEASE) of the client and it's Reactor dependencies. There have been some bug fixes, so let's see if that solves things.
here is the info from the logs . Error happens after roughly 2 hours and this is consistent. This info is from existing code , did not update anything yet. Will update and enable debug mode.
server started at 2016-09-28 15:02:19.193
Error happens at 2016-09-28 17:07:34.247
Since there's been no response on this issue in a couple of weeks, I'm going to close it. If you'd like to see it re-opened, please comment on the issue and I'll reopen it.
@mohanpindyala@s-bortolussi The latest snapshot should solve all problems around token negotiation and refresh. If you could give the snapshot a try and report back with any issues, it'd be greatly appreciated.
@ahgittin The negotiation has been updated so that a 401 triggers it every time. So, you'd expect to see a failed call, then a negotiation of a new token, and then retrying the original call. There's some fallback logic if the refresh token has expired, so possibly even two failed called before a successful retry would be expected. I'd be interested to see what's causing your hard failures though. If you set the cloudfoundry-client logger to DEBUG while it's running, it'll give detailed information about what calls are being made and what the responses are. This should be enough for me to get an idea of whether this is something systemic that we can solve.
Things were working fine. Suspended laptop. Resumed 1.5h later and refreshed browser page.
2017-01-22 03:36:30,938 [qtp469643779-21] DEBUG cloudfoundry-client.request - GET https://api.local.pcfdev.io:443/v2/organizations
2017-01-22 03:36:30,941 [qtp469643779-19] DEBUG cloudfoundry-client.request - GET https://api.local.pcfdev.io:443/v2/service_plan_visibilities
2017-01-22 03:36:30,942 [qtp469643779-20] DEBUG cloudfoundry-client.request - GET https://api.local.pcfdev.io:443/v2/service_brokers
2017-01-22 03:36:31,672 [qtp469643779-21] ERROR i.c.s.error.ExceptionController - error: {}
reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
2017-01-22 03:36:31,672 [qtp469643779-19] ERROR i.c.s.error.ExceptionController - error: {}
reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
2017-01-22 03:38:30,785 [qtp469643779-21] DEBUG cloudfoundry-client.request - GET https://api.local.pcfdev.io:443/v2/service_brokers
2017-01-22 03:38:31,710 [qtp469643779-47] DEBUG cloudfoundry-client.request - GET https://api.local.pcfdev.io:443/v2/service_brokers
Web client (where server making REST calls) stuck in loading state. Suspended laptop after a little bit. Resumed this morning, refreshed browser, and things seemed healthy.
They don't seem to say very much. Stopped the server and got these stack traces:
2017-01-23 10:18:58,343 [qtp469643779-47] ERROR i.c.s.error.ExceptionController - error: {}
reactor.core.Exceptions$ReactiveException: java.lang.InterruptedException
at reactor.core.Exceptions.propagate(Exceptions.java:218)
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:84)
at reactor.core.publisher.Mono.block(Mono.java:1173)
at io.cloudsoft.servicebroker.service.CloudFoundryController.serviceBrokers(CloudFoundryController.java:85)
at ... (suppressed spring/mvc trace)
Caused by: java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:81)
... 100 common frames omitted
and
2017-01-23 10:18:58,343 [qtp469643779-20] ERROR i.c.s.error.ExceptionController - error: {}
reactor.core.Exceptions$ReactiveException: java.lang.InterruptedException
at reactor.core.Exceptions.propagate(Exceptions.java:218)
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:84)
at reactor.core.publisher.Mono.block(Mono.java:1173)
at io.cloudsoft.servicebroker.service.CloudFoundryController.serviceBrokers(CloudFoundryController.java:85)
at ... (suppressed)
Caused by: java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:81)
... 100 common frames omitted
and
2017-01-23 10:18:58,343 [qtp469643779-21] ERROR i.c.s.error.ExceptionController - error: {}
reactor.core.Exceptions$ReactiveException: java.lang.InterruptedException
at reactor.core.Exceptions.propagate(Exceptions.java:218)
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:84)
at reactor.core.publisher.Mono.block(Mono.java:1173)
at io.cloudsoft.servicebroker.service.CloudFoundryController.serviceBrokers(CloudFoundryController.java:85)
at ... (suppressed)
Caused by: java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:81)
... 100 common frames omitted
Looks like there was some sort of lock among threads 47, 20, and 21 when the problem occurred. (Not sure what happened to 19.) So guessing the logic for refresh isn't 100% robust when the client is used concurrently in multiple threads?
(Long message with lots of evidence. Jump to the end for the TL;DR.)
With better logging and further inspection, when using our web app calling to our REST server which uses cf-java-client (current 2.3.0 snapshot), after a delay of 1h or more, we sometimes get the following:
The Connection has been aborted by the remote peer reported by the cf-java-client is not caught there, and not caught in our REST code, and returned to the browser as a 500.
{"timestamp":1485172964674,"status":500,"error":"Internal Server Error","exception":"reactor.ipc.netty.channel.ContextHandler$1","message":"Connection has been aborted by the remote peer","path":"/cf/organizations"}
This is in the following method:
@RequestMapping("/organizations")
public List<OrganizationResource> organizations() {
LOG.debug("/organizations, calling CF client to list");
try {
return cloudFoundryClient.organizations()
.list(ListOrganizationsRequest.builder().build())
.map(listOrganizationsResponse -> listOrganizationsResponse.getResources())
.block();
} finally {
LOG.debug("/organizations, finished call to CF client to list");
}
}
We observe the following in the log:
2017-01-23 12:02:44,198 DEBUG [qtp1774795940-21] i.c.s.service.CloudFoundryController /service_brokers, calling CF client to list
2017-01-23 12:02:44,198 DEBUG [qtp1774795940-45] i.c.s.service.CloudFoundryController /service_plan_visibilities, calling CF client to list
2017-01-23 12:02:44,198 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController /organizations, calling CF client to list
2017-01-23 12:02:44,240 DEBUG [qtp1774795940-21] cloudfoundry-client.request GET https://api.local.pcfdev.io:443/v2/service_brokers
2017-01-23 12:02:44,240 DEBUG [qtp1774795940-20] cloudfoundry-client.request GET https://api.local.pcfdev.io:443/v2/organizations
2017-01-23 12:02:44,240 DEBUG [qtp1774795940-45] cloudfoundry-client.request GET https://api.local.pcfdev.io:443/v2/service_plan_visibilities
2017-01-23 12:02:44,487 DEBUG [qtp1774795940-21] i.c.s.service.CloudFoundryController /service_brokers, finished call to CF client to list
2017-01-23 12:02:44,488 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController /organizations, finished call to CF client to list
2017-01-23 12:02:44,490 DEBUG [qtp1774795940-45] i.c.s.service.CloudFoundryController /service_plan_visibilities, finished call to CF client to list
2017-01-23 12:02:44,527 ERROR [qtp1774795940-45] i.c.s.error.ExceptionController error: {}
reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
2017-01-23 12:02:44,527 ERROR [qtp1774795940-20] i.c.s.error.ExceptionController error: {}
reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
2017-01-23 12:02:44,527 ERROR [qtp1774795940-21] i.c.s.error.ExceptionController error: {}
reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
Waiting a few minutes and reloading our web app triggers the following in the log (with no intervening messages):
2017-01-23 12:10:09,125 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController /service_brokers, calling CF client to list
2017-01-23 12:10:09,136 DEBUG [qtp1774795940-47] cloudfoundry-client.request GET https://api.local.pcfdev.io:443/v2/service_brokers
2017-01-23 12:10:09,164 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController /service_brokers, finished call to CF client to list
2017-01-23 12:10:09,166 ERROR [qtp1774795940-47] i.c.s.error.ExceptionController error: {}
reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
2017-01-23 12:10:09,196 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController /organizations, calling CF client to list
2017-01-23 12:10:09,198 DEBUG [qtp1774795940-20] cloudfoundry-client.request GET https://api.local.pcfdev.io:443/v2/organizations
2017-01-23 12:10:09,199 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController /service_plan_visibilities, calling CF client to list
2017-01-23 12:10:09,200 DEBUG [qtp1774795940-47] cloudfoundry-client.request GET https://api.local.pcfdev.io:443/v2/service_plan_visibilities
2017-01-23 12:10:09,206 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController /organizations, finished call to CF client to list
2017-01-23 12:10:09,212 ERROR [qtp1774795940-20] i.c.s.error.ExceptionController error: {}
reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
2017-01-23 12:10:09,302 DEBUG [reactor-http-nio-1] cloudfoundry-client.response 401 https://api.local.pcfdev.io:443/v2/service_plan_visibilities
2017-01-23 12:10:09,307 DEBUG [reactor-http-nio-1] cloudfoundry-client.request GET https://api.local.pcfdev.io:443/v2/service_plan_visibilities
2017-01-23 12:10:09,360 DEBUG [reactor-http-nio-1] cloudfoundry-client.token Negotiating using refresh token
2017-01-23 12:10:09,367 DEBUG [reactor-http-nio-1] cloudfoundry-client.request POST https://uaa.local.pcfdev.io:443/oauth/token
2017-01-23 12:10:09,593 DEBUG [reactor-http-nio-1] cloudfoundry-client.response 200 https://uaa.local.pcfdev.io:443/oauth/token
2017-01-23 12:10:09,602 DEBUG [reactor-http-nio-1] cloudfoundry-client.token Refresh Token: c89c9fafe5f64177a3afd959ecf90e16-r
2017-01-23 12:10:09,602 DEBUG [reactor-http-nio-1] cloudfoundry-client.token Access Token: bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImtleS0xIiwidHlwIjoiSldUIn0.eyJqdGkiOiJiNTMwMGJkY2FkNWU0Mjg0YTBiZWRmNTQ3ZjdlYmVmMiIsInN1YiI6ImUwYWNkZDFmLWMwMmQtNDBlMy1iNTZlLTQ1MWQxMWRkODEzNCIsInNjb3BlIjpbInJvdXRpbmcucm91dGVyX2dyb3Vwcy5yZWFkIiwiY2xvdWRfY29udHJvbGxlci5yZWFkIiwicGFzc3dvcmQud3JpdGUiLCJjbG91ZF9jb250cm9sbGVyLndyaXRlIiwib3BlbmlkIiwicm91dGluZy5yb3V0ZXJfZ3JvdXBzLndyaXRlIiwiZG9wcGxlci5maXJlaG9zZSIsInNjaW0ud3JpdGUiLCJzY2ltLnJlYWQiLCJjbG91ZF9jb250cm9sbGVyLmFkbWluIiwidWFhLnVzZXIiXSwiY2xpZW50X2lkIjoiY2YiLCJjaWQiOiJjZiIsImF6cCI6ImNmIiwiZ3JhbnRfdHlwZSI6InBhc3N3b3JkIiwidXNlcl9pZCI6ImUwYWNkZDFmLWMwMmQtNDBlMy1iNTZlLTQ1MWQxMWRkODEzNCIsIm9yaWdpbiI6InVhYSIsInVzZXJfbmFtZSI6ImFkbWluIiwiZW1haWwiOiJhZG1pbiIsInJldl9zaWciOiI5MTMzZjMzIiwiaWF0IjoxNDg0OTU5NjYwLCJleHAiOjE0ODQ5NjAyNjAsImlzcyI6Imh0dHBzOi8vdWFhLmxvY2FsLnBjZmRldi5pby9vYXV0aC90b2tlbiIsInppZCI6InVhYSIsImF1ZCI6WyJzY2ltIiwiY2xvdWRfY29udHJvbGxlciIsInBhc3N3b3JkIiwiY2YiLCJ1YWEiLCJvcGVuaWQiLCJkb3BwbGVyIiwicm91dGluZy5yb3V0ZXJfZ3JvdXBzIl19.tLRauKyT5wXDlnqbuvE84luWxNCASbVT9Td6DYKg15ocjdxDzl4_vjkJenlkZQ4-DwNZOy08sC4MQGA1sQ8EMYFS_6v8CYjJS3ao9rurR5Qn7wIvZKh0fyI46nEXcUZ8EUMUHFenRkNpaSmGS6B2r3IeGQPSHSIGsnltsH6CjJI
2017-01-23 12:10:09,615 DEBUG [reactor-http-nio-1] cloudfoundry-client.response 200 https://api.local.pcfdev.io:443/v2/service_plan_visibilities
2017-01-23 12:10:09,630 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController /service_plan_visibilities, finished call to CF client to list
This time one of the three calls to CF gets the 401 and triggers the token refresh. The webapp is now healthy (after a reload).
So it seems the problem is not any concurrency but rather that the connection to the CF server is sometimes aborted, allegedly by the remote peer (the CF server). In those cases no 401 is returned, the token not refreshed, and calls to/by the cf-java-client will continue to fail until it gets the expected 401.
There is no apparent pattern in CF returning a 401 vs a connection aborted. I am using PCF Dev on my laptop so vbox or pcf-dev could be implicated (although I'm always waiting a bit of time after resuming).
tl;dr
Things right themselves so this isn't severe. If the connection aborted is a known issue with PCF Dev only, then we could park this as a known issue with dev environments. OTOH if that behaviour happens in real CF systems then this probably should be fixed, either here or in CF, depending under what circumstances it is allowed to abort a connection.
@ahgittin I believe part of what you're seeing is due to pcfdev not handling sleeping - we have several tests that will fail with a similar time-related error if pcfdev has been slept. Naturally that's not something we'd expect to see in production (!) but it may be triggering a legitimate problem. I'll let @nebhale comment further...
A bit more info. If it's a pcf-dev sleep issue, it isn't a straightforward one.
Making one more call from the browser after the above triggers this:
2017-01-23 12:10:09,125 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController /service_brokers, calling CF client to list
2017-01-23 12:10:09,136 DEBUG [qtp1774795940-47] cloudfoundry-client.request GET https://api.local.pcfdev.io:443/v2/service_brokers
2017-01-23 12:10:09,164 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController /service_brokers, finished call to CF client to list
2017-01-23 12:10:09,166 ERROR [qtp1774795940-47] i.c.s.error.ExceptionController error: {}
reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
2017-01-23 12:10:09,196 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController /organizations, calling CF client to list
2017-01-23 12:10:09,198 DEBUG [qtp1774795940-20] cloudfoundry-client.request GET https://api.local.pcfdev.io:443/v2/organizations
2017-01-23 12:10:09,199 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController /service_plan_visibilities, calling CF client to list
And then coming back to it 30m later with no intervening sleep and doing a refresh gives:
2017-01-23 12017-01-23 12:40:30,992 DEBUG [qtp1774795940-21] i.c.s.service.CloudFoundryController /service_brokers, calling CF client to list
2017-01-23 12:40:31,005 DEBUG [qtp1774795940-21] cloudfoundry-client.request GET https://api.local.pcfdev.io:443/v2/service_brokers
2017-01-23 12:40:31,031 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController /organizations, calling CF client to list
2017-01-23 12:40:31,033 DEBUG [qtp1774795940-20] cloudfoundry-client.request GET https://api.local.pcfdev.io:443/v2/organizations
2017-01-23 12:40:31,044 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController /service_plan_visibilities, calling CF client to list
2017-01-23 12:40:31,045 DEBUG [qtp1774795940-47] cloudfoundry-client.request GET https://api.local.pcfdev.io:443/v2/service_plan_visibilities
2017-01-23 12:40:31,163 DEBUG [qtp1774795940-47] i.c.s.service.CloudFoundryController /service_plan_visibilities, finished call to CF client to list
2017-01-23 12:40:31,165 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController /organizations, finished call to CF client to list
2017-01-23 12:40:31,166 ERROR [qtp1774795940-47] i.c.s.error.ExceptionController error: {}
reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
2017-01-23 12:40:31,169 ERROR [qtp1774795940-20] i.c.s.error.ExceptionController error: {}
reactor.ipc.netty.channel.ContextHandler$1: Connection has been aborted by the remote peer
2017-01-23 12:40:31,194 DEBUG [reactor-http-nio-1] cloudfoundry-client.response 401 https://api.local.pcfdev.io:443/v2/service_brokers
2017-01-23 12:40:31,194 DEBUG [reactor-http-nio-1] cloudfoundry-client.request GET https://api.local.pcfdev.io:443/v2/service_brokers
2017-01-23 12:40:31,212 DEBUG [reactor-http-nio-1] cloudfoundry-client.token Negotiating using refresh token
2017-01-23 12:40:31,213 DEBUG [reactor-http-nio-1] cloudfoundry-client.request POST https://uaa.local.pcfdev.io:443/oauth/token
2017-01-23 12:40:31,458 DEBUG [reactor-http-nio-1] cloudfoundry-client.response 200 https://uaa.local.pcfdev.io:443/oauth/token
2017-01-23 12:40:31,460 DEBUG [reactor-http-nio-1] cloudfoundry-client.token Refresh Token: c89c9fafe5f64177a3afd959ecf90e16-r
2017-01-23 12:40:31,460 DEBUG [reactor-http-nio-1] cloudfoundry-client.token Access Token: bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImtleS0xIiwidHlwIjoiSldUIn0.eyJqdGkiOiI4YjZlZTU2MWIyZmI0ZTg0ODFkODllNjU5MjNhNjhiNyIsInN1YiI6ImUwYWNkZDFmLWMwMmQtNDBlMy1iNTZlLTQ1MWQxMWRkODEzNCIsInNjb3BlIjpbInJvdXRpbmcucm91dGVyX2dyb3Vwcy5yZWFkIiwiY2xvdWRfY29udHJvbGxlci5yZWFkIiwicGFzc3dvcmQud3JpdGUiLCJjbG91ZF9jb250cm9sbGVyLndyaXRlIiwib3BlbmlkIiwicm91dGluZy5yb3V0ZXJfZ3JvdXBzLndyaXRlIiwiZG9wcGxlci5maXJlaG9zZSIsInNjaW0ud3JpdGUiLCJzY2ltLnJlYWQiLCJjbG91ZF9jb250cm9sbGVyLmFkbWluIiwidWFhLnVzZXIiXSwiY2xpZW50X2lkIjoiY2YiLCJjaWQiOiJjZiIsImF6cCI6ImNmIiwiZ3JhbnRfdHlwZSI6InBhc3N3b3JkIiwidXNlcl9pZCI6ImUwYWNkZDFmLWMwMmQtNDBlMy1iNTZlLTQ1MWQxMWRkODEzNCIsIm9yaWdpbiI6InVhYSIsInVzZXJfbmFtZSI6ImFkbWluIiwiZW1haWwiOiJhZG1pbiIsInJldl9zaWciOiI5MTMzZjMzIiwiaWF0IjoxNDg0OTYxNDgxLCJleHAiOjE0ODQ5NjIwODEsImlzcyI6Imh0dHBzOi8vdWFhLmxvY2FsLnBjZmRldi5pby9vYXV0aC90b2tlbiIsInppZCI6InVhYSIsImF1ZCI6WyJzY2ltIiwiY2xvdWRfY29udHJvbGxlciIsInBhc3N3b3JkIiwiY2YiLCJ1YWEiLCJvcGVuaWQiLCJkb3BwbGVyIiwicm91dGluZy5yb3V0ZXJfZ3JvdXBzIl19.A5R5S4tC_s3q4kwaFp18cxnex1wM5ul87KYDD-qsvZ3ie5QW_EOeVfOAGupt3ncZzPXO7bwCvez36-U1-UxM8gMn_1ZOu7PHMjVp-gSycQfTxfzeyhecDAXl3Tn0MKAO-78KwzoR4YLNcgCaIGd-2Lia3-O-If7oC72ZbEGAxxw
2017-01-23 12:40:31,473 DEBUG [reactor-http-nio-1] cloudfoundry-client.response 200 https://api.local.pcfdev.io:443/v2/service_brokers
2017-01-23 12:40:31,487 DEBUG [qtp1774795940-21] i.c.s.service.CloudFoundryController /service_brokers, finished call to CF client to list
The webapp is unhappy due to the errors, but fine on the next refresh:
2017-01-23 12:40:42,988 DEBUG [qtp1774795940-50] i.c.s.service.CloudFoundryController /service_brokers, calling CF client to list
2017-01-23 12:40:42,989 DEBUG [qtp1774795940-50] cloudfoundry-client.request GET https://api.local.pcfdev.io:443/v2/service_brokers
2017-01-23 12:40:43,014 DEBUG [reactor-http-nio-1] cloudfoundry-client.response 200 https://api.local.pcfdev.io:443/v2/service_brokers
2017-01-23 12:40:43,015 DEBUG [qtp1774795940-50] i.c.s.service.CloudFoundryController /service_brokers, finished call to CF client to list
2017-01-23 12:40:43,076 DEBUG [qtp1774795940-21] i.c.s.service.CloudFoundryController /organizations, calling CF client to list
2017-01-23 12:40:43,078 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController /service_plan_visibilities, calling CF client to list
2017-01-23 12:40:43,079 DEBUG [qtp1774795940-20] cloudfoundry-client.request GET https://api.local.pcfdev.io:443/v2/service_plan_visibilities
2017-01-23 12:40:43,080 DEBUG [qtp1774795940-21] cloudfoundry-client.request GET https://api.local.pcfdev.io:443/v2/organizations
2017-01-23 12:40:43,096 DEBUG [reactor-http-nio-1] cloudfoundry-client.response 200 https://api.local.pcfdev.io:443/v2/service_plan_visibilities
2017-01-23 12:40:43,097 DEBUG [qtp1774795940-20] i.c.s.service.CloudFoundryController /service_plan_visibilities, finished call to CF client to list
2017-01-23 12:40:43,102 DEBUG [reactor-http-nio-1] cloudfoundry-client.response 200 https://api.local.pcfdev.io:443/v2/organizations
2017-01-23 12:40:43,115 DEBUG [qtp1774795940-21] i.c.s.service.CloudFoundryController /organizations, finished call to CF client to list
(BTW all my log messages since 2017-01-23 12:02 are included; I've just broken them in to chunks for context.)
Note that:
Our server has a single CF client and token provider bean, so I wouldn't expect the call at 2017-01-23 12:10:09,136 to fail just after token refresh.
The system has been up for 30 minutes at 12:40 so I'd be hesitant to assume that the connection aborted happening then is due to the vbox sleep, particularly as just a couple seconds later once the token is established, the same request works fine. (Also the cf CLI works fine for the same calls.)
@ahgittin Thanks for all the research; it's quite helpful. I believe that there are a couple of things going on here. In general, we'd expect to see 401s on the first request as well as when the current token has expired. It's possible to get a multiple 401s simultaneously that all resolve new tokens if requests are made concurrently (see the triple that you observe quite a lot). There's no reasonable solution to this repetitive resolution without synchronizing access to the token, so I think we're making the correct choice there.
Secondly, there is always the case that the server will terminate connections for unknown reasons. It could come from the server being over loaded (the Cloud Controller isn't the most scalable thing in the world), network devices (such as the AWS ELB) terminating connections, and even, sleeping a machine with an open connection so that the server terminates the connection, but the client doesn't know that when it starts up. In general, you probably need to handle these intermittent network failures directly as part of your code simply to be a good citizen in a massively network world; i.e. you should expect failure and account for it.
That being said, we're seeing this kind of failure a bit more than we used to and it coincides with recent changes to reactor-netty. @smaldini can you please consider this independent verification of the issues that @markpollack has reported. This is a lot of experimentation and detailed timings that may help us get to the bottom of what's going on. We'll also start taking a look at re-requesting "Connection aborted errors" for a short time by default in our code to take some of the load off users.
@nebhale@twoseat Sounds reasonable. Only thing I disagree with -- and weakly -- is whether Java consumers of the client, ie ourselves, should be doing a retry, if you were suggesting that. We give failure messages on failure. Feels like a violation of the contract if our Java code starts retrying, either we risk doing it pointlessly a lot of the time, or we put in nasty things in like if ((error instanceof reactor.ipc.netty.channel.ContextHandler$1) && error.getMessage().contains("Connection has been aborted by the remote peer")). However I totally accept JS client could retry with increasing delays, if we're really concerned about the user refreshing. :)
You've definitely fixed the major issue. I'd say close this, and maybe link to the issues you mention from @markpollack. What remains is simply a nice-to-have. Good luck resolving it and ping me if there's another build you want us to look at.
@ahgittin I absolutely am suggesting that. One of the main tenants of microservices is that you accept that failure happens and you have a strategy for dealing with it. This is most obvious example of this is the entire Circuit Breaker Pattern popularized by Netflix Hystrix. The main reason I've resisted retrying automatically so far, is that everyone has an idea of how long, how many times, what delay algorithm, etc. to use. When we talk about retry we're really hoping for something like:
Agreed that this would need to be combined with some brutal if/then to determine whether to apply that, but we can either add another version of DelayUtils.exponentialBackOffError that matches on exception types, or we can ask for a similar operator in Reactor to otherwise(Class<E>, Function<? super E, ? extends Mono<? extends T>>). At the very least, I'll ask @smaldini to give us a real exception so that it's reliably identifiable ($1 could change tomorrow just be moving code around).
Update: the call can hang; I suggest confirming that is also a known symptom of the reactor-netty issues -- and not caused by the token refresh -- before closing this issue
More detail... it seems there are at least four relevant things that we've observed happen when we call cloudFoundryClient.organizations()....block() (snippet above):
It succeeds
It returns 401 (confirmed fixed -- this is now intercepted and token refreshed and resubmitted, so we no longer see this!)
Throws "connection aborted" (due to reactor-netty ?)
Hangs
The "hang" is new, in this comment, so I'm including details. Observed with same 2.3.0.BUILD-SNAPSHOT, less often than the other behaviours but a bit more severe when it happens.
The log shows:
2017-01-23 17:02:50,981 DEBUG [qtp1675239551-48] i.c.s.service.CloudFoundryController /organizations, calling CF client to list
2017-01-23 17:02:50,982 DEBUG [qtp1675239551-48] cloudfoundry-client.request GET https://api.local.pcfdev.io:443/v2/organizations
The corresponding completion messages are never displayed, implying that the thread is hung.
jstack shows this:
"qtp1675239551-48" #48 prio=5 os_prio=31 tid=0x00007fd543075800 nid=0x8f03 waiting on condition [0x0000700011375000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000778cc7040> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:81)
at reactor.core.publisher.Mono.block(Mono.java:1173)
at io.cloudsoft.servicebroker.service.CloudFoundryController.organizations(CloudFoundryController.java:159)
(removed the rest, spring MVC)
I've made multiple curl calls to trigger this code, and each generates a new such hung thread as above.
Making other calls seems to refresh things sufficiently that this call subsequently works, so it is recoverable, although there will be a thread leak as the hung threads such as the above do not seem to be resolved after success. (Maybe they will be on session expiry / logout, if you track calls and run some scavenger.)
@nebhale Thanks for clarifying. In our case the java is a fairly minimal proxy. Our JS client needs to be able to handle failures no matter how much smarts we put in our proxy, ideally with the exponential back-off retry you suggest (though currently it just reports failure to the user who can retry, which is fine for now) so marginal benefit putting it also in the java. If our java tracked state I'd agree with you.
It would be nice if cf-java-client provides a retry mechanism (with fairly low timeout bounds and/or configurable, though in both cases it seems a clunky contract) or richer failure semantics (definitely no $1!). But clearly the best fix is to sort out whatever is going on between reactor-netty and CF server -- especially as it is sometimes leaking threads as above, and it looks like the cf CLI is not encountering these issues. (Implementing retry is just going to compound the problem here! :P )
@nebhale now after delay JWT expiration is the cause of an aborted connection, client side, with the latest current snapshot of the client (PCF Dev has been suspended and resumed) -- not sure if this was contributing to the previous aborts or not. it is new today and the only thing changed is the cf-java-client version (LMK if this should be separate issue). HTH.
2017-01-27 17:08:35,550 ERROR [HttpClient] Error processing connection. Requesting close the channel
io.jsonwebtoken.ExpiredJwtException: JWT expired at 2017-01-27T15:16:18Z. Current time: 2017-01-27T17:08:35Z, a difference of 6737534 milliseconds. Allowed clock skew: 0 milliseconds.
at io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:385)
at io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:481)
at io.jsonwebtoken.impl.DefaultJwtParser.parseClaimsJwt(DefaultJwtParser.java:514)
at org.cloudfoundry.reactor.tokenprovider.AbstractUaaTokenProvider.parseToken(AbstractUaaTokenProvider.java:147)
at org.cloudfoundry.reactor.tokenprovider.AbstractUaaTokenProvider.extractAccessToken(AbstractUaaTokenProvider.java:128)
@ahgittin That's the addition of some client-side logging that should only happen when the token is first issued. Is it possible that the clock inside the pcfdev instances has skewed? Either way, I'll protect against this exception to ensure that the debug logging doesn't kill anything.
Activity
cf-gitbot commentedon Sep 28, 2016
We have created an issue in Pivotal Tracker to manage this:
https://www.pivotaltracker.com/story/show/131326071
The labels on this github issue will be updated when the story is started.
twoseat commentedon Sep 29, 2016
@mohanpindyala I suspect this is because your token has expired. Is 'a while' the same period every time?
nebhale commentedon Sep 29, 2016
It does certainly appear that the token being presented to the API is invalid. What does your
TokenProviderconfiguration look like?mohanpindyala commentedon Sep 29, 2016
there you go . Also we are using operations object everywhere and not underlying client
nebhale commentedon Sep 29, 2016
Can you please turn the logger
cloudfoundry-clienttoDEBUG. This should start generating a lot of information about the requests made to the server including negotiation of the token. It'd be helpful to get the contents of these outputs (you can filter out everything else that's application specific) from the start of the application (and the first negotiation of a token) all the way to the moment that the token is not longer accepted.nebhale commentedon Sep 29, 2016
In addition to that logging, please update to the latest version (
2.0.1.RELEASE) of the client and it's Reactor dependencies. There have been some bug fixes, so let's see if that solves things.mohanpindyala commentedon Sep 29, 2016
oh thats nice , let me update to 2.0.1 release and see if that helps
mohanpindyala commentedon Sep 29, 2016
here is the info from the logs . Error happens after roughly 2 hours and this is consistent. This info is from existing code , did not update anything yet. Will update and enable debug mode.
server started at 2016-09-28 15:02:19.193
Error happens at 2016-09-28 17:07:34.247
nebhale commentedon Nov 2, 2016
Since there's been no response on this issue in a couple of weeks, I'm going to close it. If you'd like to see it re-opened, please comment on the issue and I'll reopen it.
CF client Invalid auth token error
s-bortolussi commentedon Nov 4, 2016
@nebhale after upgrading to lastest release 2.0.1.RELEASE, go the same exception.
Using these versions for java client
and log activation
See following logs including my comments (error seems to be caused by a refresh token expiration.)
Notice, that refresh token expiration return HTTP 401 error but no exception seems to be raised.
Latter, when performing action using cf java client, got the following exception :
Refresh token expiration
Improved Token Negotiation
nebhale commentedon Jan 18, 2017
@mohanpindyala @s-bortolussi The latest snapshot should solve all problems around token negotiation and refresh. If you could give the snapshot a try and report back with any issues, it'd be greatly appreciated.
5 remaining items
nebhale commentedon Jan 21, 2017
@ahgittin The negotiation has been updated so that a
401triggers it every time. So, you'd expect to see a failed call, then a negotiation of a new token, and then retrying the original call. There's some fallback logic if the refresh token has expired, so possibly even two failed called before a successful retry would be expected. I'd be interested to see what's causing your hard failures though. If you set thecloudfoundry-clientlogger toDEBUGwhile it's running, it'll give detailed information about what calls are being made and what the responses are. This should be enough for me to get an idea of whether this is something systemic that we can solve.ahgittin commentedon Jan 23, 2017
@nebhale Here are logs from the REST server.
Things were working fine. Suspended laptop. Resumed 1.5h later and refreshed browser page.
Web client (where server making REST calls) stuck in loading state. Suspended laptop after a little bit. Resumed this morning, refreshed browser, and things seemed healthy.
They don't seem to say very much. Stopped the server and got these stack traces:
and
and
Looks like there was some sort of lock among threads
47,20, and21when the problem occurred. (Not sure what happened to19.) So guessing the logic for refresh isn't 100% robust when the client is used concurrently in multiple threads?ahgittin commentedon Jan 23, 2017
(Long message with lots of evidence. Jump to the end for the TL;DR.)
With better logging and further inspection, when using our web app calling to our REST server which uses
cf-java-client(current2.3.0snapshot), after a delay of 1h or more, we sometimes get the following:The
Connection has been aborted by the remote peerreported by thecf-java-clientis not caught there, and not caught in our REST code, and returned to the browser as a500.{"timestamp":1485172964674,"status":500,"error":"Internal Server Error","exception":"reactor.ipc.netty.channel.ContextHandler$1","message":"Connection has been aborted by the remote peer","path":"/cf/organizations"}This is in the following method:
We observe the following in the log:
Waiting a few minutes and reloading our web app triggers the following in the log (with no intervening messages):
This time one of the three calls to CF gets the
401and triggers the token refresh. The webapp is now healthy (after a reload).So it seems the problem is not any concurrency but rather that the connection to the CF server is sometimes aborted, allegedly by the remote peer (the CF server). In those cases no
401is returned, the token not refreshed, and calls to/by thecf-java-clientwill continue to fail until it gets the expected401.There is no apparent pattern in CF returning a
401vs aconnection aborted. I am using PCF Dev on my laptop so vbox or pcf-dev could be implicated (although I'm always waiting a bit of time after resuming).tl;dr
Things right themselves so this isn't severe. If the
connection abortedis a known issue with PCF Dev only, then we could park this as a known issue with dev environments. OTOH if that behaviour happens in real CF systems then this probably should be fixed, either here or in CF, depending under what circumstances it is allowed to abort a connection.twoseat commentedon Jan 23, 2017
@ahgittin I believe part of what you're seeing is due to pcfdev not handling sleeping - we have several tests that will fail with a similar time-related error if pcfdev has been slept. Naturally that's not something we'd expect to see in production (!) but it may be triggering a legitimate problem. I'll let @nebhale comment further...
ahgittin commentedon Jan 23, 2017
A bit more info. If it's a pcf-dev sleep issue, it isn't a straightforward one.
Making one more call from the browser after the above triggers this:
And then coming back to it 30m later with no intervening sleep and doing a refresh gives:
The webapp is unhappy due to the errors, but fine on the next refresh:
(BTW all my log messages since
2017-01-23 12:02are included; I've just broken them in to chunks for context.)Note that:
2017-01-23 12:10:09,136to fail just after token refresh.12:40so I'd be hesitant to assume that theconnection abortedhappening then is due to the vbox sleep, particularly as just a couple seconds later once the token is established, the same request works fine. (Also thecfCLI works fine for the same calls.)HTH
nebhale commentedon Jan 23, 2017
@ahgittin Thanks for all the research; it's quite helpful. I believe that there are a couple of things going on here. In general, we'd expect to see
401s on the first request as well as when the current token has expired. It's possible to get a multiple401s simultaneously that all resolve new tokens if requests are made concurrently (see the triple that you observe quite a lot). There's no reasonable solution to this repetitive resolution without synchronizing access to the token, so I think we're making the correct choice there.Secondly, there is always the case that the server will terminate connections for unknown reasons. It could come from the server being over loaded (the Cloud Controller isn't the most scalable thing in the world), network devices (such as the AWS ELB) terminating connections, and even, sleeping a machine with an open connection so that the server terminates the connection, but the client doesn't know that when it starts up. In general, you probably need to handle these intermittent network failures directly as part of your code simply to be a good citizen in a massively network world; i.e. you should expect failure and account for it.
That being said, we're seeing this kind of failure a bit more than we used to and it coincides with recent changes to
reactor-netty. @smaldini can you please consider this independent verification of the issues that @markpollack has reported. This is a lot of experimentation and detailed timings that may help us get to the bottom of what's going on. We'll also start taking a look at re-requesting "Connection aborted errors" for a short time by default in our code to take some of the load off users.ahgittin commentedon Jan 23, 2017
@nebhale @twoseat Sounds reasonable. Only thing I disagree with -- and weakly -- is whether Java consumers of the client, ie ourselves, should be doing a retry, if you were suggesting that. We give failure messages on failure. Feels like a violation of the contract if our Java code starts retrying, either we risk doing it pointlessly a lot of the time, or we put in nasty things in like
if ((error instanceof reactor.ipc.netty.channel.ContextHandler$1) && error.getMessage().contains("Connection has been aborted by the remote peer")). However I totally accept JS client could retry with increasing delays, if we're really concerned about the user refreshing. :)You've definitely fixed the major issue. I'd say close this, and maybe link to the issues you mention from @markpollack. What remains is simply a nice-to-have. Good luck resolving it and ping me if there's another build you want us to look at.
nebhale commentedon Jan 23, 2017
@ahgittin I absolutely am suggesting that. One of the main tenants of microservices is that you accept that failure happens and you have a strategy for dealing with it. This is most obvious example of this is the entire Circuit Breaker Pattern popularized by Netflix Hystrix. The main reason I've resisted retrying automatically so far, is that everyone has an idea of how long, how many times, what delay algorithm, etc. to use. When we talk about retry we're really hoping for something like:
Agreed that this would need to be combined with some brutal if/then to determine whether to apply that, but we can either add another version of
DelayUtils.exponentialBackOffErrorthat matches on exception types, or we can ask for a similar operator in Reactor tootherwise(Class<E>, Function<? super E, ? extends Mono<? extends T>>). At the very least, I'll ask @smaldini to give us a real exception so that it's reliably identifiable ($1could change tomorrow just be moving code around).ahgittin commentedon Jan 23, 2017
Update: the call can hang; I suggest confirming that is also a known symptom of the
reactor-nettyissues -- and not caused by the token refresh -- before closing this issueMore detail... it seems there are at least four relevant things that we've observed happen when we call
cloudFoundryClient.organizations()....block()(snippet above):The "hang" is new, in this comment, so I'm including details. Observed with same
2.3.0.BUILD-SNAPSHOT, less often than the other behaviours but a bit more severe when it happens.The log shows:
The corresponding completion messages are never displayed, implying that the thread is hung.
jstackshows this:I've made multiple
curlcalls to trigger this code, and each generates a new such hung thread as above.Making other calls seems to refresh things sufficiently that this call subsequently works, so it is recoverable, although there will be a thread leak as the hung threads such as the above do not seem to be resolved after success. (Maybe they will be on session expiry / logout, if you track calls and run some scavenger.)
ahgittin commentedon Jan 23, 2017
@nebhale Thanks for clarifying. In our case the java is a fairly minimal proxy. Our JS client needs to be able to handle failures no matter how much smarts we put in our proxy, ideally with the exponential back-off retry you suggest (though currently it just reports failure to the user who can retry, which is fine for now) so marginal benefit putting it also in the java. If our java tracked state I'd agree with you.
It would be nice if
cf-java-clientprovides a retry mechanism (with fairly low timeout bounds and/or configurable, though in both cases it seems a clunky contract) or richer failure semantics (definitely no$1!). But clearly the best fix is to sort out whatever is going on betweenreactor-nettyand CF server -- especially as it is sometimes leaking threads as above, and it looks like thecfCLI is not encountering these issues. (Implementing retry is just going to compound the problem here! :P )ahgittin commentedon Jan 27, 2017
@nebhale now after delay JWT expiration is the cause of an aborted connection, client side, with the latest current snapshot of the client (PCF Dev has been suspended and resumed) -- not sure if this was contributing to the previous aborts or not. it is new today and the only thing changed is the
cf-java-clientversion (LMK if this should be separate issue). HTH.nebhale commentedon Jan 27, 2017
@ahgittin That's the addition of some client-side logging that should only happen when the token is first issued. Is it possible that the clock inside the
pcfdevinstances has skewed? Either way, I'll protect against this exception to ensure that the debug logging doesn't kill anything.Protect Against Any Token Parsing Failure
upgrade cf-java-client to 2.4.0.RELEASE