Improve status check for pulling an image through docker-swarm #337

hawky-4s- · 2015-10-06T11:52:28Z

This improves the PullImageCmd for docker-swarm because docker-swarm returns a different status line than the normal docker client.
This fix only checks if the last status line returns successfully when pulling through docker-swarm, which means that the pull on the last node was successful. It does not mean the image pull on all swarm nodes went ok.
I do not know how to check this, because to be fully swarm compatible it would be needed to read the last x lines where x is the number of swarm nodes and check them all if the result is successful.

This change is

marcuslinke · 2015-10-06T15:08:35Z

I do not know how to check this, because to be fully swarm compatible it would be needed to read the last x lines where x is the number of swarm nodes and check them all if the result is successful.

This should be possible by inherit PullImageResultCallback and override awaitSuccess() then. WDYT?

hawky-4s- · 2015-10-09T12:40:15Z

I followed your suggestion but only changed the implementation of the PullImageResultCallback. It is now able to differentiate between Docker Swarm and Docker Client and adjust it's behavior accordingly. I did this so it is transparent to the user of the api if he's talking to a swarm or client without knowing it in advance.

WDYT?

marcuslinke · 2015-10-09T18:56:56Z

src/main/java/com/github/dockerjava/core/command/PullImageResultCallback.java

How this should work. The first item that doesn't indicate a succesfull pull will lead to pull error.

Swarm works like so. When you issue a pull cmd, it returns following responses as PullResponseItem for each node in the swarm:
21:38:57.269 DEBUG c.g.d.c.c.PullImageResultCallback - PullResponseItem[stream=<null>,status=Pulling ci1.camunda.loc:5000/camunda-ci-postgresql:9.4...,progressDetail=ResponseItem.ProgressDetail[current=0,total=0,start=0],progress=<null>,id=ci7.camunda.loc,from=<null>,time=0,errorDetail=<null>,error=<null>]
When a pull is successfully for a swarm node following PullResponseItem is returned:
21:39:04.513 DEBUG c.g.d.c.c.PullImageResultCallback - PullResponseItem[stream=<null>,status=Pulling ci1.camunda.loc:5000/camunda-ci-postgresql:9.4... : downloaded,progressDetail=ResponseItem.ProgressDetail[current=0,total=0,start=0],progress=<null>,id=ci3.camunda.loc,from=<null>,time=0,errorDetail=<null>,error=<null>]
There are only these two responses for each swarm node when a pull request is successfull.

My code will first collect all PullResponseItems ordered by node id as key. So only the latest PullResponseItem of each node will be stored, previous ones overwritten.
When the PullImageCmd is finished and awaitSuccess method continues to evaluate the success of the pull, it will check the last result of each node. When one of those results indicates a failed pull, it will throw an exception with all nodes where the pull failed.

Ok. Now i got it. Thanks!

KostyaSha · 2015-10-09T19:56:53Z

Interesting, how this all can be tested... DinD?

marcuslinke · 2015-10-09T20:05:25Z

Shouldn't it be possible to start a swarm manager and a swarm node as containers at the docker host and handle this single docker host as a swarm? This would be the easiest solution...

KostyaSha · 2015-10-09T20:10:48Z

Doesn't swarm do pull on two hosts at once?

hawky-4s- · 2015-10-09T20:27:44Z

You can start a swarm manager and point it to the local docker server like
docker run -it --rm -p 2377:2375 swarm manage nodes://<node_ip>:2375
Then configure <node_ip>:2377 as endpoint for docker-java.

hawky-4s- · 2015-10-09T20:28:07Z

@KostyaSha
Swarm pulls on as many nodes as you have concurrently.

KostyaSha · 2015-10-09T20:28:50Z

Swarm pulls on as many nodes as you have concurrently.

It what i mean about tests complexity. (Just curious)

hawky-4s- · 2015-10-09T20:30:34Z

Yes, but you could also just start a single swarm master configured to use the same node where it runs as swarm member with the command I provided above.

hawky-4s- · 2015-10-09T20:33:51Z

Example debug output from running the pull cmd against a swarm using docker-java:

DEBUG c.g.d.c.c.PullImageResultCallback - PullResponseItem[stream=<null>,status=Pulling ci1.camunda.loc:5000/camunda-ci-postgresql:9.4...,progressDetail=ResponseItem.ProgressDetail[current=0,total=0,start=0],progress=<null>,id=ci7.camunda.loc,from=<null>,time=0,errorDetail=<null>,error=<null>]
DEBUG c.g.d.c.c.PullImageResultCallback - PullResponseItem[stream=<null>,status=Pulling ci1.camunda.loc:5000/camunda-ci-postgresql:9.4...,progressDetail=ResponseItem.ProgressDetail[current=0,total=0,start=0],progress=<null>,id=ci3.camunda.loc,from=<null>,time=0,errorDetail=<null>,error=<null>]
DEBUG c.g.d.c.c.PullImageResultCallback - PullResponseItem[stream=<null>,status=Pulling ci1.camunda.loc:5000/camunda-ci-postgresql:9.4...,progressDetail=ResponseItem.ProgressDetail[current=0,total=0,start=0],progress=<null>,id=ci6.camunda.loc,from=<null>,time=0,errorDetail=<null>,error=<null>]
DEBUG c.g.d.c.c.PullImageResultCallback - PullResponseItem[stream=<null>,status=Pulling ci1.camunda.loc:5000/camunda-ci-postgresql:9.4...,progressDetail=ResponseItem.ProgressDetail[current=0,total=0,start=0],progress=<null>,id=ci4.camunda.loc,from=<null>,time=0,errorDetail=<null>,error=<null>]
DEBUG c.g.d.c.c.PullImageResultCallback - PullResponseItem[stream=<null>,status=Pulling ci1.camunda.loc:5000/camunda-ci-postgresql:9.4... : downloaded,progressDetail=ResponseItem.ProgressDetail[current=0,total=0,start=0],progress=<null>,id=ci4.camunda.loc,from=<null>,time=0,errorDetail=<null>,error=<null>]
DEBUG c.g.d.c.c.PullImageResultCallback - PullResponseItem[stream=<null>,status=Pulling ci1.camunda.loc:5000/camunda-ci-postgresql:9.4... : downloaded,progressDetail=ResponseItem.ProgressDetail[current=0,total=0,start=0],progress=<null>,id=ci3.camunda.loc,from=<null>,time=0,errorDetail=<null>,error=<null>]
DEBUG c.g.d.c.c.PullImageResultCallback - PullResponseItem[stream=<null>,status=Pulling ci1.camunda.loc:5000/camunda-ci-postgresql:9.4... : downloaded,progressDetail=ResponseItem.ProgressDetail[current=0,total=0,start=0],progress=<null>,id=ci7.camunda.loc,from=<null>,time=0,errorDetail=<null>,error=<null>]
DEBUG c.g.d.c.c.PullImageResultCallback - PullResponseItem[stream=<null>,status=Pulling ci1.camunda.loc:5000/camunda-ci-postgresql:9.4... : downloaded,progressDetail=ResponseItem.ProgressDetail[current=0,total=0,start=0],progress=<null>,id=ci6.camunda.loc,from=<null>,time=0,errorDetail=<null>,error=<null>]

KostyaSha · 2016-09-28T23:01:02Z

#711 on the way, then your changes should be picked.

KostyaSha · 2016-09-29T22:18:01Z

Trying to pick into swarm branch. Help is welcome.

KostyaSha · 2017-05-05T02:31:40Z

picking via #716

djl197 mentioned this pull request Oct 8, 2015

Unable to pull image using 'Pull Once - Update always' scheme jenkinsci/docker-plugin#339

Closed

hawky-4s- force-pushed the swarm-pull branch 3 times, most recently from 1c7dcca to 06382cc Compare October 9, 2015 12:37

marcuslinke reviewed Oct 9, 2015
View reviewed changes

Improve PullImageCmd for pulling an image through docker swarm.

a39e52d

hawky-4s- force-pushed the swarm-pull branch from 06382cc to a39e52d Compare October 9, 2015 19:54

KostyaSha added the area/swarm-classic label Nov 10, 2015

marcuslinke mentioned this pull request Nov 30, 2015

Implement Node property in Inspect container response introduces by swarm #383

Merged

marcuslinke mentioned this pull request Jan 30, 2016

Made the client swarm-compatible #443

Closed

marcuslinke added this to the 3.1.0 milestone Jan 30, 2016

marcuslinke mentioned this pull request Feb 12, 2016

swarm support #395

Closed

KostyaSha mentioned this pull request Sep 29, 2016

Improve PullImageCmd for pulling an image through docker swarm. #716

Closed

KostyaSha closed this May 5, 2017

Improve status check for pulling an image through docker-swarm #337

Improve status check for pulling an image through docker-swarm #337

Uh oh!

Conversation

hawky-4s- commented Oct 6, 2015 • edited by marcuslinke Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marcuslinke commented Oct 6, 2015

Uh oh!

hawky-4s- commented Oct 9, 2015

Uh oh!

marcuslinke Oct 9, 2015

Choose a reason for hiding this comment

Uh oh!

hawky-4s- Oct 9, 2015

Choose a reason for hiding this comment

Uh oh!

marcuslinke Oct 9, 2015

Choose a reason for hiding this comment

Uh oh!

KostyaSha commented Oct 9, 2015

Uh oh!

marcuslinke commented Oct 9, 2015

Uh oh!

KostyaSha commented Oct 9, 2015

Uh oh!

hawky-4s- commented Oct 9, 2015

Uh oh!

hawky-4s- commented Oct 9, 2015

Uh oh!

KostyaSha commented Oct 9, 2015

Uh oh!

hawky-4s- commented Oct 9, 2015

Uh oh!

hawky-4s- commented Oct 9, 2015

Uh oh!

KostyaSha commented Sep 28, 2016

Uh oh!

KostyaSha commented Sep 29, 2016

Uh oh!

KostyaSha commented May 5, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hawky-4s- commented Oct 6, 2015 •

edited by marcuslinke

Loading