You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Change the bulk of our tests to "true" unit tests, and kill the fake database #2476
Now that we have embedded postgres, the fake database is not needed for quick local environments, and the main purpose is testing.
However, we have to do some nasty stuff, (#2473) to keep it working, and because it is not postgres, it can lead us into a false sense of security about whether our code actually works.
But, we do need a suite of tests that run fast, and our postgres tests are slow.
CLI tests, by and large, shouldn't be calling the server at all, and instead be mocking out the codersdk. That will make them simpler (because you don't need a bunch of preamble to get the DB into the state you want), and super fast. It will also make is easier to test things other than the happy path like errors.
Coderd tests can and should mock out the database interface for most cases, for the same reasons.
We still will want a couple integration tests that go all the way from the CLI -> API -> Database, but there will only be a few and we could use postgres for that.
Once we're in that state, we could consider killing the fake database.
it can lead us into a false sense of security about whether our code actually works
While it can, we require PostgreSQL checks to pass in CI before merging, so I'm not sure this could ever hit production.
CLI tests, by and large, shouldn't be calling the server at all, and instead be mocking out the codersdk. That will make them simpler (because you don't need a bunch of preamble to get the DB into the state you want), and super fast. It will also make is easier to test things other than the happy path like errors.
Coderd tests can and should mock out the database interface for most cases, for the same reasons.
I think this issue would involve trying this to see if it nets out simpler and more reliable. I'm not convinced we wouldn't just end up with mocks that would distribute issues rather than centralize them. We have an expected set of functionality for each database call so another option might be recreating the fake.
Speed is the real thing here from my perspective. It takes ~20s to run all tests for Coder with make test. Compared to PostgreSQL, which takes a few minutes. If we were able to get these comparable, I'd happily swap over.
changed the title [-]change the bulk of our tests to "true" unit tests, and kill the fake database[/-][+]Change the bulk of our tests to "true" unit tests, and kill the fake database[/+]on Aug 24, 2022
Activity
misskniss commentedon Aug 9, 2022
@kylecarbs and @spikecurtis agree there should be more discussion on this before we start.
kylecarbs commentedon Aug 9, 2022
While it can, we require PostgreSQL checks to pass in CI before merging, so I'm not sure this could ever hit production.
I think this issue would involve trying this to see if it nets out simpler and more reliable. I'm not convinced we wouldn't just end up with mocks that would distribute issues rather than centralize them. We have an expected set of functionality for each database call so another option might be recreating the fake.
Speed is the real thing here from my perspective. It takes ~20s to run all tests for Coder with
make test. Compared to PostgreSQL, which takes a few minutes. If we were able to get these comparable, I'd happily swap over.[-]change the bulk of our tests to "true" unit tests, and kill the fake database[/-][+]Change the bulk of our tests to "true" unit tests, and kill the fake database[/+]kylecarbs commentedon Oct 21, 2022
I will close this for now because we don't have a clear path to improve this.
Running with real PostgreSQL takes a significantly longer time (4x as long), so we probably don't want to swap immediately.