e2e testing - #141

It looks like you have removed all of the logic I had for ensuring that permissions on volume directories hasn't changed: https://github.com/sourcegraph/deploy-sourcegraph-docker/pull/141/files#diff-b584b34513d743c4570b69a661750e14L37-L64

We need to add this back, we can't remove it because this is a really important property to test for. The customer using this, https://app.hubspot.com/contacts/2762526/company/407948923/ does not have the ./ensure-volume.sh $VOLUME 100 invocations and that must be communicated through the upgrading docs as I said here: https://github.com/sourcegraph/deploy-sourcegraph-docker/pull/141/files#diff-b584b34513d743c4570b69a661750e14L50-L51

I have a PR with the information pending here

All of the deploy-.sh scripts that require a volume now envoke ensure-volume.sh. Is this customer running a specific fork that doesn't include that? If so, wouldn't it make more sense for them to have that script?

I am not sure how to avoid this confusion, but see the README of this repository for an explanation of what pure-docker deployments actually are:

The goal is that anyone using a container management platform other than Kubernetes (Netflix's Titus, Apache's Mesos, etc.) would be able to use this repository as a reference for how to deploy Sourcegraph.

The point being the customer in question does NOT run the scripts in this repository, and any time there are changes in volume permissions etc. they need to know about those precisely because it means they need to be able to adapt those to their deployment in their own way. It's not a fork - it's the way they were intended to consume this repository (as a reference)

The point being: we need these lines to still exist in this test somewhere, removing them means we can't in good faith release pure docker to this customer and the test becomes effectively useless

The most obvious solution to me here is to remove the deploy.sh call above and move all of this logic (deployment and other smoke tests) directly into smoke-test.sh - I am happy to do that now if you're OK with that

Side note: It's not really clear what should be in shell commands here vs. in smoke-test.sh

Personally, I am inclined to put the heavy duty tasks in the scripts ie: setting up volumes, testing and then execute those scripts in the shell commands.

Previously when I've used this harness I'd been a quick and easy way to just comment out a script in shell commands if I've wanted to run the vagrant box then ssh in to manually test or tweak things before committing them.

FYI I've moved the setup and configuration part of the test process into a new script, and the testing itself is still done with smoke-test.sh

My 2c, I would think of it the same way you think about the CI pipeline yaml.

• Steps are listed in the yaml
• Each step is unit of work that does not know about the others

I won't block this further because I am content with this as-is, so feel free to merge.

But the organization here still doesn't make much sense to me.

• setup.sh doesn't install dependencies, the Vagrantfile does
• The main property being tested by these tests is in fact the static permissions on volume directories, but we've placed that in a setup.sh file - why is testing done in a setup file? Should this be renamed to perms-test.sh?
• smoke-test.sh relies on the work that setup.sh is doing by setting up the volume directories - when this test fails in smoke-test.sh because a container fails to come up with e.g. wrong permissions, someone will need to find setup.sh and fix permissions there. This seems not super intuitive to me.
• servers.yaml is calling deploy.sh and is "responsible for starting the deployment" but smoke-test.sh is responsible for actually ensuring the containers came up OK?

Maybe I am missing something, but the structure that would make a lot more sense to me is:

• servers.yaml only calls smoke-test.sh, nothing else.
• everything in setup.sh currently goes into smoke-test.sh
• dependency installation goes into an external script setup.sh (and thus Vagrantfile becomes something we can just copy for other test suites in the future.)