Skip to content

Fix autoscale without breaking manual scaling - #108

Closed
kingdonb wants to merge 22 commits into
teamhephy:masterfrom
kingdonb:kb-fix-autoscale-take-3
Closed

kingdonb wants to merge 22 commits into
teamhephy:masterfrom
kingdonb:kb-fix-autoscale-take-3

Conversation

@kingdonb

@kingdonb kingdonb commented Aug 6, 2019

Copy link
Copy Markdown
Member

This PR is a WIP

We've been testing changes to make the Autoscale functionality via HPA work again.

Right now it looks like this image (quay.io/kingdonb/controller:kb-fix-autoscale-take-5) is successfully permitting a manual scale as well as an HPA to be created, like:

deis autoscale:set cmd -a testing-hephy --min=2 --max=5 --cpu-percent=10

The concession we had to make is that we don't know what has changed with manual scaling in apps/v1 deployment, so we have left it at extensions/v1beta1 API for deployments.

The autoscaler/v1 HorizontalPodAutoscaler will only find the Deployment as a match if it is also provided with a scaleTargetRef.apiVersion selector that matches. I added that here, and HPA now succeeds, assuming you have metrics-server properly configured so the HPA can get pod metrics to make scale decisions.

I also tested the manual deis ps:scale -a testing-hephy cmd=1 behavior, which was broken by the last attempt we made to upgrade to autoscaler/v1. We should make any reasonable efforts to upgrade the deployment to apps/v1, but that is not necessary in order to fix the autoscale feature (without unfortunately also breaking manual scaling.) This PR seems to do both 👍!

The other frankly unacceptable concession I had to make was in rootfs/api/models/app.py where the HPA is created, I have hardcoded 'apiVersion': 'extensions/v1beta1' as I'm not sure how the App resource is supposed to find out what the Deployment's api_version ought to be.

We discussed maybe adding an API version service that you can use from any module to determine what API versions are in use for what objects, but there is more discussion that needs to happen before it's clear what that piece ought to look like.

Also, since upgrading a resource's API version means necessarily destroying the old resource, we are going to need a bigger plan when it comes to upgrading deployments to apps/v1, since that step appears to be unavoidable, we might need to write code so that a cluster can recognize when these deployment upgrades are needed, and to handle them in a reasonable way. I'm not sure how this can be done without potentially causing apps to have downtime.

(It should be perfectly safe to delete and recreate HPAs though, so for the upgrade guide, maybe we give that a little mention and this can go to press relatively as-is in the next release.)

@kingdonb kingdonb changed the title Fix autoscale take 3..4..5... Fix autoscale without breaking manual scaling Aug 6, 2019
Cryptophobia and others added 22 commits August 6, 2019 14:12
fix(controller): revert release.check_image_access for now
This is basically just manually re-applying the
reverted changes from teamhephy#86
This functionality now lives in KubeHTTPClient, so
it is unnecessary to mutate the reported k8s API
version in the deployment resource.
Refactoring for new KubeHTTPClient.version() behavior
Upgrading to apps/v1 requires changes to the way that Scale action is
called, and as we haven't understood this change yet, we will depend on
the older version of Deployment api to remain available until we do.

includes these squashed commits:
* WIP: this commit is probably wrong
* WIP: this seems more likely to work
* WIP: hardcode extensions/v1beta1 here for now

There is no "App.api_version" attribute and I'm not sure where this
information should come from

* fixup: line was too long for the style guide
@kingdonb
kingdonb force-pushed the kb-fix-autoscale-take-3 branch from 4cf2e01 to ce6f29b Compare August 6, 2019 20:14
@kingdonb

kingdonb commented Aug 6, 2019

Copy link
Copy Markdown
Member Author

I rebased this with some --onto and --rebase-merges magic, so it wouldn't include any reverse-merges back from master into the wip branch.

@kingdonb

kingdonb commented Aug 6, 2019

Copy link
Copy Markdown
Member Author

We can probably also take this to a new branch and do some commit squashing to improve the history, for the changelog.

@ess

ess commented Aug 6, 2019

Copy link
Copy Markdown
Contributor

Tested this on an EKS cluster running 1.13, verified as working.

@Cryptophobia
Cryptophobia self-requested a review August 8, 2019 14:35
@Cryptophobia

Copy link
Copy Markdown
Member

@kingdonb , I would really like to improve the commit messages. I'm going to take your commits and put them on top of #106 and merge them there. I will test again and merge everything once the tests pass. We probably want to squash some of these commits as well, the ones in the beginning (reverts).

@Cryptophobia Cryptophobia left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm! This will be merged in #106. Thank you for taking care of this during the testing @kingdonb !!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants