Skip to content

Change the bulk of our tests to "true" unit tests, and kill the fake database #2476

Closed as not planned

Description

@spikecurtis

Problem

Now that we have embedded postgres, the fake database is not needed for quick local environments, and the main purpose is testing.

However, we have to do some nasty stuff, (#2473) to keep it working, and because it is not postgres, it can lead us into a false sense of security about whether our code actually works.

But, we do need a suite of tests that run fast, and our postgres tests are slow.

CLI tests, by and large, shouldn't be calling the server at all, and instead be mocking out the codersdk. That will make them simpler (because you don't need a bunch of preamble to get the DB into the state you want), and super fast. It will also make is easier to test things other than the happy path like errors.

Coderd tests can and should mock out the database interface for most cases, for the same reasons.

We still will want a couple integration tests that go all the way from the CLI -> API -> Database, but there will only be a few and we could use postgres for that.

Once we're in that state, we could consider killing the fake database.

Activity

  1. misskniss commented on Aug 9, 2022

    @misskniss

    @kylecarbs and @spikecurtis agree there should be more discussion on this before we start.

  2. kylecarbs commented on Aug 9, 2022

    @kylecarbs
    Member

    it can lead us into a false sense of security about whether our code actually works

    While it can, we require PostgreSQL checks to pass in CI before merging, so I'm not sure this could ever hit production.

    CLI tests, by and large, shouldn't be calling the server at all, and instead be mocking out the codersdk. That will make them simpler (because you don't need a bunch of preamble to get the DB into the state you want), and super fast. It will also make is easier to test things other than the happy path like errors.

    Coderd tests can and should mock out the database interface for most cases, for the same reasons.

    I think this issue would involve trying this to see if it nets out simpler and more reliable. I'm not convinced we wouldn't just end up with mocks that would distribute issues rather than centralize them. We have an expected set of functionality for each database call so another option might be recreating the fake.

    Speed is the real thing here from my perspective. It takes ~20s to run all tests for Coder with make test. Compared to PostgreSQL, which takes a few minutes. If we were able to get these comparable, I'd happily swap over.

  3. changed the title [-]change the bulk of our tests to "true" unit tests, and kill the fake database[/-] [+]Change the bulk of our tests to "true" unit tests, and kill the fake database[/+] on Aug 24, 2022
  4. kylecarbs commented on Oct 21, 2022

    @kylecarbs
    Member

    I will close this for now because we don't have a clear path to improve this.

    Running with real PostgreSQL takes a significantly longer time (4x as long), so we probably don't want to swap immediately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions