Skip to content

cli core services setup and migration#873

Merged
kushalmalani merged 41 commits into
mainfrom
mluo/cli-core-services-migration
Nov 23, 2022
Merged

cli core services setup and migration#873
kushalmalani merged 41 commits into
mainfrom
mluo/cli-core-services-migration

Conversation

@missing1984
Copy link
Copy Markdown
Contributor

@missing1984 missing1984 commented Nov 14, 2022

Description

This PR migrated 3 endpoints to consume core services like UI does today.

  1. listOrganizations
  2. getSelfUser
  3. createUserInvite

3 commands are impacted by this change

  1. astro organization list. - consume listOrganizations endpoint
  2. astro user invite. - consume createUserInvite endpoint
  3. astro login - consume both listOrganizations and getSelfUser endpoints

Code details

Openapi codegen and core service client at astro-client-core/*

  • github.com/deepmap/oapi-codegen is brought in to generate the openapi client for core services. all generated code are in astro-client-core/api.gen.go.
  • astro-client-core/client.go wrap up the integration of generated client and our context (setting auth headers)

cmd/* layer change

  • initialized astrocore.CoreClient from root and pass down to all cloud commands

cloud/* layer change.

  • modified organization package to consume core service only.
  • modified user package to consume core service only.
  • modified auth package to consume core service getSelf and list organization.

Other notable change

  • orgShortName is a required argument in order to call core services. currently we don't have it persisted in config. a migration.go module is introduced that backfill the organization_short_name field. the migration function will run at the end of the cloud setup function.
  • astro-client-core came with generated mocks. All relevant unit tests are switched over to use the new mocked client.

also, this is my first golang PR so don't hesitate to comment out dumb mistake..

🎟 Issue(s)

https://github.com/astronomer/astro/issues/2247

🧪 Functional Testing

List the functional testing steps to confirm this feature or fix.

📸 Screenshots

Add screenshots to illustrate the validity of these changes.

📋 Checklist

  • Rebased from the main (or release if patching) branch (before testing)
  • Ran make test before taking out of draft
  • Ran make lint before taking out of draft
  • Added/updated applicable tests
  • Tested against Astro-API (if necessary).
  • Tested against Houston-API and Astronomer (if necessary).
  • Communicated to/tagged owners of respective clients potentially impacted by these changes.
  • Updated any related documentation

Comment thread cloud/user/user.go
return err
}
derivedOrganizationID, err := getOrganizationID(client)
ctx, err = context.GetCurrentContext()
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jemishp as we discussed, i simply read the org from context and send invite on behalf of the org without doing a reverse lookup. Let me know if this will be a problem

Copy link
Copy Markdown
Contributor

@jemishp jemishp Nov 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it have an issue where the inviter's CLI does not have the correct OrgID in its org as the inviter may have been removed from the org?

Would core api catch that? I guess we cant test it out and refine as well.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

core api will error out in this case, also i think this is more align with user's exceptions which invite is sent on behalf of current org. We are good here.

Comment thread cmd/cloud/migration.go
return err
}
// backfill OrganizationShortName
if (ctx.OrganizationShortName == "") && (ctx.Organization != "") {
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here is the code that backfill organization_short_name in config

Copy link
Copy Markdown
Contributor

@jemishp jemishp Nov 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just curious why would organization_short_name not be filled in because we already call c.SetOrganizationContext() with it if the user comes in via checkUserSession() or checkToken(), or via checkAPIKeys()

Copy link
Copy Markdown
Contributor Author

@missing1984 missing1984 Nov 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because the backfill path in those place are optional.

  1. checkAPIKeys will only run in the context of api key.
  2. checkToken will not backfill if the token is valid as we only backfill short name in login path.

so a "catch all" migration step feels safer. I think in the future we will run into migration scenario as well so having a migration step will be helpful.

Comment thread cmd/cloud/setup.go
@codecov
Copy link
Copy Markdown

codecov Bot commented Nov 15, 2022

Codecov Report

Base: 87.29% // Head: 87.32% // Increases project coverage by +0.03% 🎉

Coverage data is based on head (af9b084) compared to base (6556478).
Patch coverage: 88.19% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #873      +/-   ##
==========================================
+ Coverage   87.29%   87.32%   +0.03%     
==========================================
  Files         108      109       +1     
  Lines        9326     9366      +40     
==========================================
+ Hits         8141     8179      +38     
  Misses        697      697              
- Partials      488      490       +2     
Impacted Files Coverage Δ
context/context.go 82.05% <ø> (ø)
cmd/cloud/setup.go 58.70% <71.42%> (+2.24%) ⬆️
cmd/cloud/migration.go 78.94% <78.94%> (ø)
cmd/auth.go 89.09% <83.33%> (ø)
config/context.go 77.58% <87.50%> (+1.88%) ⬆️
cloud/organization/organization.go 78.02% <92.00%> (+1.32%) ⬆️
cloud/auth/auth.go 81.94% <93.75%> (+0.67%) ⬆️
cloud/user/user.go 100.00% <100.00%> (ø)
cloud/workspace/workspace.go 84.14% <100.00%> (ø)
cmd/cloud/organization.go 92.94% <100.00%> (ø)
... and 6 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Comment thread cloud/auth/auth.go
break
}
}
err = c.SetOrganizationContext(activeOrg.Id, activeOrg.ShortName)
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is guard against a rare use case where user's login org is different than the org he can access. We always treat AuthZ is the source of truth of user org relationship

@missing1984
Copy link
Copy Markdown
Contributor Author

This is ready for review. i am adding new unit tests now.

@missing1984 missing1984 changed the title draft: cli core services migration cli core services setup and migration Nov 15, 2022
Comment thread cloud/auth/auth.go Outdated
Comment thread cloud/auth/auth.go
missing1984 and others added 2 commits November 17, 2022 18:17
Co-authored-by: Jemish Patel <37133965+jemishp@users.noreply.github.com>
Co-authored-by: Jemish Patel <37133965+jemishp@users.noreply.github.com>
Comment thread cloud/auth/auth.go
Comment thread cloud/user/user.go
Comment thread cloud/user/user_test.go Outdated
Comment thread cloud/auth/auth_test.go
Copy link
Copy Markdown
Contributor

@kushalmalani kushalmalani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a couple of comments..Code looks ok otherwise. We will be waiting to merge this in until 1.8 pre release is cut.

Copy link
Copy Markdown
Contributor

@neel-astro neel-astro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left few minor comments, otherwise lgtm

Comment thread astro-client-core/client.go Outdated
Comment thread astro-client-core/client.go Outdated
Comment thread astro-client-core/client.go
Comment thread cloud/user/user.go Outdated
Comment thread Makefile
Comment thread Makefile
@sunkickr
Copy link
Copy Markdown
Contributor

could you make sure that the refresh tokens code still works. Code is located here:

func refresh(refreshToken string, authConfig astro.AuthConfig) (TokenResponse, error) {

@missing1984
Copy link
Copy Markdown
Contributor Author

missing1984 commented Nov 22, 2022

could you make sure that the refresh tokens code still works. Code is located here:

func refresh(refreshToken string, authConfig astro.AuthConfig) (TokenResponse, error) {

@sunkickr How to test it? I modified the exp date in config and rerun command -- a new token is fetched and saved.

but i ran into an issue that the command which triggers the renew does not return anything.

➜  astro-cli git:(main) ./astro workspace list  ->>> this command triggers renew but nothing is output.
 NAME     ID     
➜  astro-cli git:(main) ./astro workspace list  ->>> rerun displayed the right information.
 NAME                                                 ID                              
 Registry Daedalus                                    cl9egxx2b303242cx16ey2eyt6      
 MatthiasSecTesting                                   cla6wzgkd137147ahygq1v2p7nd     
 new ws                                               ckyblko4p57591f35ln2s56lo       
 Workspace 1                                          ckv19qvd2390411dzb71wxbbzn  

this is happening on main as well so i don't think it's introduced by my change.

@kushalmalani kushalmalani merged commit b1c6033 into main Nov 23, 2022
@kushalmalani kushalmalani deleted the mluo/cli-core-services-migration branch November 23, 2022 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants