[Azure/Loadbalancer] Creating an internal load balancer on Azure Kubernetes Service (AKS).

Mình thực hiện apply service với IP internal.

https://learn.microsoft.com/en-us/azure/aks/internal-lb?tabs=set-service-annotations

apiVersion: v1
kind: Service
metadata:
 name: internal-app
 annotations:
   service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
 type: LoadBalancer
 ports:
 - port: 80
 selector:
   app: internal-app

Service đã được tạo nhưng external IP vẫn Pending:

root@work-space-u20:~/azure-cloud/aks_vnet# kubectl get service
NAME           TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
internal-app   LoadBalancer   172.17.188.247   <pending>     80:31732/TCP   32m
kubernetes     ClusterIP      172.17.0.1       <none>        443/TCP        17h

Nếu bạn thực hiện describe service thì thấy các lỗi dưới.

Events:
  Type     Reason                  Age                From                Message
  ----     ------                  ----               ----                -------
  Normal   EnsuringLoadBalancer    32s (x4 over 68s)  service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  32s (x4 over 68s)  service-controller  Error syncing load balancer: failed to ensure load balancer: GET http://localhost:7788/subscriptions/7f8e5acb-a937-4163-80b1-f874c3a4d62c/resourceGroups/nimtechnology/providers/Microsoft.Network/virtualNetworks/elearning/subnets/elearning-subnet-1
--------------------------------------------------------------------------------
RESPONSE 403: 403 Forbidden
ERROR CODE: AuthorizationFailed
--------------------------------------------------------------------------------
{
  "error": {
    "code": "AuthorizationFailed",
    "message": "The client '67805159-f222-4ecc-9e21-9c87f067ed02' with object id '67805159-f222-4ecc-9e21-9c87f067ed02' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/subnets/read' over scope '/subscriptions/7f8e5acb-a937-4163-80b1-f874c3a4d62c/resourceGroups/nimtechnology/providers/Microsoft.Network/virtualNetworks/elearning/subnets/elearning-subnet-1' or the scope is invalid. If access was recently granted, please refresh your credentials."
  }
}
--------------------------------------------------------------------------------

External Traffic Policy:  Cluster
Events:
  Type     Reason                  Age                From                Message
  ----     ------                  ----               ----                -------
  Normal   EnsuringLoadBalancer    14s (x3 over 29s)  service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  14s (x3 over 29s)  service-controller  Error syncing load balancer: failed to ensure load balancer: GET http://localhost:7788/subscriptions/7f8e5acb-a937-4163-80b1-f874c3a4d62c/resourceGroups/nimtechnology/providers/Microsoft.Network/virtualNetworks/elearning/subnets/elearning-subnet-1
--------------------------------------------------------------------------------
RESPONSE 403: 403 Forbidden
ERROR CODE: AuthorizationFailed
--------------------------------------------------------------------------------
{
  "error": {
    "code": "AuthorizationFailed",
    "message": "The client '67805159-f222-4ecc-9e21-9c87f067ed02' with object id '67805159-f222-4ecc-9e21-9c87f067ed02' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/subnets/read' over scope '/subscriptions/7f8e5acb-a937-4163-80b1-f874c3a4d62c/resourceGroups/nimtechnology/providers/Microsoft.Network/virtualNetworks/elearning/subnets/elearning-subnet-1' or the scope is invalid. If access was recently granted, please refresh your credentials."
  }
}

Để kiểm tra client ID này là của ai thì bạn có thể kiểm tra bằng lên bên dưới.

root@work-space-u20:~/azure-cloud/aks_vnet# az ad sp show --id 67805159-f222-4ecc-9e21-9c87f067ed02
{
  "@odata.context": "https://graph.microsoft.com/v1.0/$metadata#servicePrincipals/$entity",
  "accountEnabled": true,
  "addIns": [],
  "alternativeNames": [
    "isExplicit=False",
    "/subscriptions/7f8e5acb-a937-4163-80b1-f874c3a4d62c/resourcegroups/nimtechnology/providers/Microsoft.ContainerService/managedClusters/terraform-aks-elearning-cluster"
  ],
  "appDescription": null,
  "appDisplayName": null,
  "appId": "a40551dc-b98d-40cf-8c6b-d9d19ef99aad",
  "appOwnerOrganizationId": null,
  "appRoleAssignmentRequired": false,
  "appRoles": [],
  "applicationTemplateId": null,
  "createdDateTime": "2025-05-12T10:20:35Z",
  "deletedDateTime": null,
  "description": null,
  "disabledByMicrosoftStatus": null,
  "displayName": "terraform-aks-elearning-cluster",
  "homepage": null,
  "id": "67805159-f222-4ecc-9e21-9c87f067ed02",
  "info": null,
  "keyCredentials": [],
  "loginUrl": null,
  "logoutUrl": null,
  "notes": null,
  "notificationEmailAddresses": [],
  "oauth2PermissionScopes": [],
  "passwordCredentials": [],
  "preferredSingleSignOnMode": null,
  "preferredTokenSigningKeyThumbprint": null,
  "replyUrls": [],
  "resourceSpecificApplicationPermissions": [],
  "samlSingleSignOnSettings": null,
  "servicePrincipalNames": [
    "a40551dc-b98d-40cf-8c6b-d9d19ef99aad",
    "https://identity.azure.net/6rPoKQaw+9ew40ibYojGX6XgWsclqyNuivczR9LfQ6g="
  ],
  "servicePrincipalType": "ManagedIdentity",
  "signInAudience": null,
  "tags": [],
  "tokenEncryptionKeyId": null,
  "verifiedPublisher": {
    "addedDateTime": null,
    "displayName": null,
    "verifiedPublisherId": null
  }
}

Đây là các subnet của mình

Mô hình hiện tại thì mình cấp Subnet cho các node pool của AKS:

Ở đây chúng ta cần tạo permission cho AKS được đọc vào Subnet:

# Assign the Network Contributor role to the AKS managed identity on each subnet

resource "azurerm_role_assignment" "aks_network_contributor" {
 principal_id         = azurerm_kubernetes_cluster.aks_cluster.identity[0].principal_id
 role_definition_name = "Network Contributor"
 scope                = <Subnet_ID>
}

Sau khi apply permission chúng ta thấy service đã lấy được IP của subnet đặt cho service loadbalancer