Revert "Revert "Enable load balancer slow start"" - #51418
Merged
sureshc merged 3 commits intoApr 27, 2023
Conversation
…nable Slow Start.
cat5inthecradle
approved these changes
Apr 21, 2023
…ents in production.
cat5inthecradle
self-requested a review
April 26, 2023 21:17
cat5inthecradle
approved these changes
Apr 26, 2023
sureshc
deleted the
revert-51253-revert-51105-enable-load-balancer-slow-start
branch
April 27, 2023 18:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reverts #51253
Our previous attempt to deploy this change - #51105 - failed because we were using an incompatible load balancing algorithm. Re-attempting this change.
Enable a 300 second slow start (successfully tested via manual configuration change in production) to reduce the likelihood of performance issues when releasing to production during peak usage time periods. Switch load balancing algorithms (something we've tested manually in production during peak usage to ensure no impact on performance) because we were using an algorithm that is incompatible with Slow Start.
https://codedotorg.atlassian.net/browse/INF-864
Testing story
Our managed test environment provisions a load balancer, but does not have an Auto Scaling Group implemented, so no new "Front End" (web application server) EC2 Instances will be provisioned on our managed test server when this change reaches that environment, so this feature will not be exercised until it reaches production.
We could provision an adhoc with the
Full Stack(auto scaling, load balancer, RDS database, etc.) enabled, but we have not been successful in getting those to provision lately, and also if we pushed a new commit to this feature branch to simulate a production release, a longstanding limitation in our adhoc provisioning logic, we re-provision the entire stack as a single EC2 Instance environment (no CloudFront, no load balancer, no auto scaling group, no RDS cluster, etc.).Tested in production via manual configuration change. In one case, 300 second Slow Start period resulted in a release to production during peak usage where there was NO performance degradation. With 120 seconds Slow Start period, there was at least one deploy where performance deteriorated during new application server launch. See notes in Slack.
Deployment strategy
Possibly release as a 2nd production deploy later in the work day to minimize risk.
Follow-up work
Privacy
Security
Caching
PR Checklist: