Fix AKS Scaling (Cluster Autoscaler)

[mithunshanbhag]

We're investigating the following options for AKS scaling:

1. ACI VIRTUAL NODES

Status

Currently BLOCKED.

Where

• The code changes are in mithun/hpa2 branch. (see PR microsoft/ContosoTraders#78)
• The deployment is in the contoso-traders test subscription.

Change Description

• Redeployed AKS cluster via bicep template from mithun/hpa2 branch, which has the Azure CNI network policy (instead of the default kubenet policy).
• Had to manually modify AKS's vnet to create a new subnet aci-subnet with address space 10.255.0.0/16.
• Tethered it to existing AKS cluster using az aks enable-addons (full instructions here).
• Applied the Deployment.yaml manifest from mithun/hpa2 branch, which has the nodeSelector, tolerations changes to configure pods to only run in virtual nodes.

Issue Details

The pods (configured to run in ACI virtual nodes) are stuck in waiting state.

The logs only show that an active endpoint is not being detected for the services / ingress

Hypothesis

• Could have something to do with the fact that we switched over to Azure CNI network policy instead of the default kubenet policy.
• Could have something to do with the nodeSelector, tolerations changes made in the Deployment.yaml file to configure pod to only run in virtual nodes.

2. CLUSTER AUTOSCALER

Status

Currently INVESTIGATING

Where

• The changes are in my fork in [mithun/enable-autoscal](mithunshanbhag:mithun/cluster-autoscaler) branch (See PR [DRAFT] enable cluster autoscaling #81)
• Deployed in Jithin's MSDN subscription.

Change Description

• Enable autoscaling with minCount: 1 and maxCount: 10

Issue Details

• Load test has a high failure rate. This issue is being tracked separately in Load Test: Fix API issues identified from load testing contosotraders-cloudtesting#3

  

• The pods are also not scaling out (this could be related to above issue).

  

Hypothesis

Currently none, still investigating.

Misc Notes

Ingress controller was stuck in PENDING state for a few minutes after provisioning. Then automatically went to OK state.