Skip to content

Revert "Revert "Enable load balancer slow start"" - #51418

Merged
sureshc merged 3 commits into
stagingfrom
revert-51253-revert-51105-enable-load-balancer-slow-start
Apr 27, 2023
Merged

Revert "Revert "Enable load balancer slow start""#51418
sureshc merged 3 commits into
stagingfrom
revert-51253-revert-51105-enable-load-balancer-slow-start

Conversation

@sureshc

@sureshc sureshc commented Apr 19, 2023

Copy link
Copy Markdown
Contributor

Reverts #51253

Our previous attempt to deploy this change - #51105 - failed because we were using an incompatible load balancing algorithm. Re-attempting this change.

Enable a 300 second slow start (successfully tested via manual configuration change in production) to reduce the likelihood of performance issues when releasing to production during peak usage time periods. Switch load balancing algorithms (something we've tested manually in production during peak usage to ensure no impact on performance) because we were using an algorithm that is incompatible with Slow Start.

https://codedotorg.atlassian.net/browse/INF-864

Testing story

Our managed test environment provisions a load balancer, but does not have an Auto Scaling Group implemented, so no new "Front End" (web application server) EC2 Instances will be provisioned on our managed test server when this change reaches that environment, so this feature will not be exercised until it reaches production.

We could provision an adhoc with the Full Stack (auto scaling, load balancer, RDS database, etc.) enabled, but we have not been successful in getting those to provision lately, and also if we pushed a new commit to this feature branch to simulate a production release, a longstanding limitation in our adhoc provisioning logic, we re-provision the entire stack as a single EC2 Instance environment (no CloudFront, no load balancer, no auto scaling group, no RDS cluster, etc.).

Tested in production via manual configuration change. In one case, 300 second Slow Start period resulted in a release to production during peak usage where there was NO performance degradation. With 120 seconds Slow Start period, there was at least one deploy where performance deteriorated during new application server launch. See notes in Slack.

Deployment strategy

Possibly release as a 2nd production deploy later in the work day to minimize risk.

Follow-up work

Privacy

Security

Caching

PR Checklist:

  • Tests provide adequate coverage
  • Privacy and Security impacts have been assessed
  • Code is well-commented
  • New features are translatable or updates will not break translations
  • Relevant documentation has been added or updated
  • User impact is well-understood and desirable
  • Pull Request is labeled appropriately
  • Follow-up work items (including potential tech debt) are tracked and linked

@sureshc
sureshc requested a review from a team as a code owner April 19, 2023 18:59
@cat5inthecradle
cat5inthecradle self-requested a review April 26, 2023 21:17
@sureshc
sureshc merged commit 33218ea into staging Apr 27, 2023
@sureshc
sureshc deleted the revert-51253-revert-51105-enable-load-balancer-slow-start branch April 27, 2023 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants