Skip to content

feat: Implement unified pagination and add template versions support - #1308

Merged
mafredri merged 21 commits into
mainfrom
mafredri/feat-paginate-template-versions-endpoint
May 10, 2022
Merged

mafredri merged 21 commits into
mainfrom
mafredri/feat-paginate-template-versions-endpoint

Conversation

@mafredri

@mafredri mafredri commented May 5, 2022 •

Copy link
Copy Markdown
Member

This PR adds support for pagination to the /api/v2/templates/:id/versions endpoint whilst breaking out some of the common functionality that can be shared between endpoints that will support pagination. As such, the /api/v2/users endpoint also received a small refactor.

After a discussion with the team, I settled on using embedded structs for the pagination options. We may want to consider using pointers &codersdk.Pagination{} for being explicit when they're not set, but since the zero values work well it should compose better in more use cases (e.g. req := codersdk.MyRequest{}; req.Limit = 10). Among other API designs considered were functional options and and optional arguments inside optional structs.

I decided to omit support for search in the template versions endpoint since I was uncertain if this will be needed. If we want to support it, we may want to consider support for defining the column/field to be searched (e.g. should search be applied to name, description or both).

Questions:

@Emyrk Currently using both offset and after_id are supported, as per your design. What are your thoughts about splitting these up? We could for instance create two separate types: type PaginateOffset struct and type PaginateID struct, an endpoint can choose to support either or both by embedding (example):

type MyEndpointRequest struct {
        MyParam int
        PaginateOffset
        PaginateID
}

Benefits of splitting them up would be to allow an endpoint to use a lesser feature set, but maybe there's a use-case for always supporting both?

Misc thoughts:

  • In codersdk it would make sense to me to rename all .*Request structs to .*Params since they define the parameters that can be used for the Client method(s). My reasoning is that we're doing a request by invoking the client method, the params only define the arguments.
  • Should we start to consider adding support for ordering by a specified column as well? (This would be left up to future exploration)

Fixes #53

@mafredri
mafredri requested a review from a team as a code owner May 5, 2022 12:18
@mafredri mafredri self-assigned this May 5, 2022
@codecov

codecov Bot commented May 5, 2022 •

Copy link
Copy Markdown

Codecov Report

Merging #1308 (aab400c) into main (dc115b8) will increase coverage by 0.01%.
The diff coverage is 59.48%.

@@            Coverage Diff             @@
##             main    #1308      +/-   ##
==========================================
+ Coverage   66.81%   66.82%   +0.01%     
==========================================
  Files         284      282       -2     
  Lines       18633    18493     -140     
  Branches      235      235              
==========================================
- Hits        12449    12358      -91     
+ Misses       4903     4871      -32     
+ Partials     1281     1264      -17     
Flag Coverage Δ
unittest-go-macos-latest 54.07% <52.58%> (+0.13%) ⬆️
unittest-go-postgres- 65.59% <54.31%> (+0.07%) ⬆️
unittest-go-ubuntu-latest 56.36% <52.58%> (-0.03%) ⬇️
unittest-go-windows-2022 ?
unittest-js 73.67% <ø> (ø)
Impacted Files Coverage Δ
scripts/apitypings/main.go 0.00% <0.00%> (ø)
coderd/pagination.go 51.35% <51.35%> (ø)
coderd/templates.go 64.24% <76.92%> (+1.95%) ⬆️
coderd/users.go 61.44% <88.88%> (+0.81%) ⬆️
coderd/database/queries.sql.go 77.90% <100.00%> (+0.06%) ⬆️
codersdk/client.go 59.74% <100.00%> (ø)
codersdk/pagination.go 100.00% <100.00%> (ø)
codersdk/templates.go 60.37% <100.00%> (ø)
codersdk/users.go 63.79% <100.00%> (-0.62%) ⬇️
cli/configssh.go 62.58% <0.00%> (-6.48%) ⬇️
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dc115b8...aab400c. Read the comment docs.

Comment thread coderd/templates_test.go Resolved

@kylecarbs kylecarbs left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Just two minor comments 😎

Comment thread coderd/httpapi/httpapi.go Resolved
Comment thread coderd/templates.go Resolved
Comment thread codersdk/pagination.go Resolved
Comment thread site/src/api/typesGenerated.ts Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@BrunoQuaresma just want to make sure you see this and get a chance to discuss it if needed

Comment thread site/src/api/typesGenerated.ts Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean we send either after_id or limit and offset? I think typescript will complain if we leave anything out of this type, so can we send 0 or empty string for the ones we're not using?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@presleyp The way this is (currently) implemented means you can set any combination of them (including setting all or none). I don't know if we want to keep it that way though.

@Emyrk do you know if there's a way we can make the generator express the above? (E.g. would setting ? all of these, like readonly after_id?: string, work?)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I'm concerned that if the backend expects "just omit the ones you don't want to use" but the frontend enforces "these are all required" then we might have trouble sending well formed requests.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I agree, that would be super inconvenient! I see pointers and a few other null types are marked as optional by the TypeScript generator, but not others. I could take a look at improving that. Perhaps using the json omitempty option (Go specific) could also be marked optional in TypeScript.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think both after_id and offset make sense together. And, I'll argue that we should drop offset entirely in favor of after_id.

The problem with offset is that it is not safe to race conditions that add or delete rows. Say we have {a0, a1, a2, b0, b1, b2} as our data. I paginate with limit=3 offset=0 and get {a0, a1, a2}. Now someone else deletes a1. Then I call paginate with limit=3 offset=3. I get {b1, b2}!

You might argue that in some use cases, it doesn't matter if we duplicate or omit data, so why not give both options? The problem is that it's really hard to be sure we've communicated the danger about offset to potential users of this API. They might not read the docstrings. This API might end up being offered to end users outside of Coder. That offset is so tempting because the code is easier to write than after_id, but we're setting users up to shoot themselves in the foot.

@mafredri mafredri May 5, 2022 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with this. It's also why I suggested separating the options (offset and after_id) into two structs in my PR description. But since adding both was deliberate (#1057 (comment)) I don't think we should remove it in this PR (at least not for the users endpoint). Personally I don't see much of a usability difference between the two and would prefer only having after_id (or another solution entirely).

Comment thread codersdk/pagination.go Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand the use case for using both AfterID and Offset. I would have thought they should be mutually exclusive.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, I'd be in favor of dropping offset if there's no use-case for it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found some context in an older PR: #1057 (comment)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@presleyp @BrunoQuaresma Can the frontend just use cursor based pagination? How much harder would it be to support over offset

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't argue for offset-based pagination over cursor-based pagination based on how hard it is to implement on the frontend, but...we don't need pagination on the frontend in CE so we are currently ignoring it, sending no limit or offset params, which I believe works because the offset-based pagination defaults are the same as if it wasn't paginated. Could cursor-based pagination be similarly ignored?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor-based pagination is similarly ignored if not set in the implementation I'm reading.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opened up an issue/feature for this: #1358

Comment thread site/src/api/typesGenerated.ts Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think both after_id and offset make sense together. And, I'll argue that we should drop offset entirely in favor of after_id.

The problem with offset is that it is not safe to race conditions that add or delete rows. Say we have {a0, a1, a2, b0, b1, b2} as our data. I paginate with limit=3 offset=0 and get {a0, a1, a2}. Now someone else deletes a1. Then I call paginate with limit=3 offset=3. I get {b1, b2}!

You might argue that in some use cases, it doesn't matter if we duplicate or omit data, so why not give both options? The problem is that it's really hard to be sure we've communicated the danger about offset to potential users of this API. They might not read the docstrings. This API might end up being offered to end users outside of Coder. That offset is so tempting because the code is easier to write than after_id, but we're setting users up to shoot themselves in the foot.

Comment thread codersdk/pagination.go Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This locks us into having a UUID for every database table that we want to paginate. Have we fully considered the consequences?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had the same thought, but at least all existing tables seem to be using UUID for id.

@coder/backend Does anyone else have input on this? Steven seems to be on PTO so can't check in with him.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can easily be changed to a string or something if we ever want to support different ID types in the future so I think it's fine for now

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if the after_id doesn't exist? Either because user modified a URL, or because someone deleted the row?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No results, which is acceptable in the former scenario. As for the latter, good catch. Not sure if this has been thought of.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opened up an issue for this #1357 (also relevant #1358).

@deansheather deansheather left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mafredri
mafredri force-pushed the mafredri/feat-paginate-template-versions-endpoint branch from 04b3b1f to 13fc406 Compare May 10, 2022 07:30
…gs (#1318)

* feat: Add support for json omitempty to apitypings

* feat: Express embedded structs via extends in TypeScript

* Handle unembedding via json field in apitypings

* Add scripts/apitypings/main.go to Makefile
@mafredri
mafredri enabled auto-merge (squash) May 10, 2022 07:42
@mafredri
mafredri merged commit 2d3dc43 into main May 10, 2022
@mafredri
mafredri deleted the mafredri/feat-paginate-template-versions-endpoint branch May 10, 2022 07:44
@mafredri mafredri changed the title feat: Implement pagination for template versions feat: Implement unified pagination and add template versions support May 10, 2022
@misskniss misskniss added this to the V2 Beta milestone May 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Paginate template history

7 participants