02.29 kubernetes系列教程(十九)使用metric-server讓HPA彈性伸縮運行

1. 監控架構概述

kubernetes監控指標大體可以分為兩類：核心監控指標和自定義指標，核心監控指標是kubernetes內置穩定可靠監控指標，早期由heapster完成，現由metric-server實現；自定義指標用於實現核心指標的擴展，能夠提供更豐富的指標支持，如應用狀態指標，自定義指標需要通過Aggregator和k8s api集成，當前主流通過promethues實現。

監控指標用途：

2. metric-server架構和安裝

2. 1 metric-server簡介

Metrics Server is a cluster-wide aggregator of resource usage data. Resource metrics are used by components like kubectl top and the Horizontal Pod Autoscaler to scale workloads. To autoscale based upon a custom metric, you need to use the Prometheus Adapter Metric-server是一個集群級別的資源指標收集器，用於收集資源指標數據

The Metrics API

未來將能夠支持指標聚合，如max最大值，min最小值，95th峰值，以及自定義時間窗口，如1h，1d，1w等。

2.2 metric-server架構

monitoring_architecture.png

監控架構分兩部分內容：核心監控(圖白色部分)和自定義監控（圖藍色部分）

1、 核心監控實現

2、 自定義監控實現

2.3 metric-server部署

1、獲取metric-server安裝文件，當前具有兩個版本：1.7和1.8+，kubernetes1.7版本安裝1.7的metric-server版本，kubernetes 1.8後版本安裝metric server 1.8+版本

<code>[root@node-1 ~]# git clone https://github.com/kubernetes-sigs/metrics-server.git/<code>

2、部署metric-server，部署1.8+版本

<code>[root@node-1 metrics-server]# kubectl apply -f deploy/1.8+/clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader createdclusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator createdrolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader createdapiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io createdserviceaccount/metrics-server createddeployment.apps/metrics-server createdservice/metrics-server createdclusterrole.rbac.authorization.k8s.io/system:metrics-server createdclusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created/<code>

核心的配置文件是metrics-server-deployment.yaml，metric-server以Deployment的方式部署在集群中，j鏡像k8s.gcr.io/metrics-server-amd64:v0.3.6需要提前下載好，其對應的安裝文件內容如下：

<code>---apiVersion: v1kind: ServiceAccountmetadata: name: metrics-server namespace: kube-system---apiVersion: apps/v1kind: Deploymentmetadata: name: metrics-server namespace: kube-system labels: k8s-app: metrics-serverspec: selector: matchLabels: k8s-app: metrics-server template: metadata: name: metrics-server labels: k8s-app: metrics-server spec: serviceAccountName: metrics-server volumes: # mount in tmp so we can safely use from-scratch images and/or read-only containers - name: tmp-dir emptyDir: {} containers: - name: metrics-server image: k8s.gcr.io/metrics-server-amd64:v0.3.6 args: - --cert-dir=/tmp - --secure-port=4443ExternalIP ports: - name: main-port containerPort: 4443 protocol: TCP securityContext: readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 1000 imagePullPolicy: Always volumeMounts: - name: tmp-dir mountPath: /tmp nodeSelector: beta.kubernetes.io/os: linux/<code>

3、檢查metric-server部署的情況,查看metric-server的Pod已部署成功

<code>[root@node-1 1.8+]# kubectl get deployments metrics-server -n kube-system NAME READY UP-TO-DATE AVAILABLE AGEmetrics-server 1/1 1 1 2m49s[root@node-1 1.8+]# kubectl get pods -n kube-system metrics-server-67db467b7b-5xf8x NAME READY STATUS RESTARTS AGEmetrics-server-67db467b7b-5xf8x 1/1 Running 0 3m/<code>

實際此時metric-server並不能使用，使用kubectl top node <node-id>查看會提示Error from server (NotFound): nodemetrics.metrics.k8s.io "node-1" not found類似的報錯，查看metric-server的pod的日誌信息，顯示如下：/<node-id>

<code>[root@node-1 1.8+]# kubectl logs metrics-server-67db467b7b-5xf8x -n kube-system -fI1230 11:34:10.905500 1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)I1230 11:34:11.527346 1 secure_serving.go:116] Serving securely on [::]:4443E1230 11:35:11.552067 1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:node-1: unable to fetch metrics from Kubelet node-1 (node-1): Get https://node-1:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup node-1 on 10.96.0.10:53: no such host, unable to fully scrape metrics from source kubelet_summary:node-3: unable to fetch metrics from Kubelet node-3 (node-3): Get https://node-3:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup node-3 on 10.96.0.10:53: no such host, unable to fully scrape metrics from source kubelet_summary:node-2: unable to fetch metrics from Kubelet node-2 (node-2): Get https://node-2:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup node-2 on 10.96.0.10:53: no such host]/<code>

4、上述的報錯信息提示pod中通過DNS無法解析主機名，可以通過在pod中定義hosts文件或告知metric-server優先使用IP的方式通訊，修改metric-server的deployment配置文件，修改如下並重新應用配置

修改metric-server部署配置文件

5、應用metric-server部署文件後重新生成一個pod，日誌中再次查看提示另外一個報錯信息

<code>[root@node-1 1.8+]# kubectl logs metrics-server-f54f5d6bf-s42rc -n kube-system -fI1230 11:45:26.615547 1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)I1230 11:45:27.043723 1 secure_serving.go:116] Serving securely on [::]:4443E1230 11:46:27.065274 1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:node-2: unable to fetch metrics from Kubelet node-2 (10.254.100.102): Get https://10.254.100.102:10250/stats/summary?only_cpu_and_memory=true: x509: cannot validate certificate for 10.254.100.102 because it doesn't contain any IP SANs, unable to fully scrape metrics from source kubelet_summary:node-1: unable to fetch metrics from Kubelet node-1 (10.254.100.101): Get https://10.254.100.101:10250/stats/summary?only_cpu_and_memory=true: x509: cannot validate certificate for 10.254.100.101 because it doesn't contain any IP SANs, unable to fully scrape metrics from source kubelet_summary:node-3: unable to fetch metrics from Kubelet node-3 (10.254.100.103): Get https://10.254.100.103:10250/stats/summary?only_cpu_and_memory=true: x509: cannot validate certificate for 10.254.100.103 because it doesn't contain any IP SANs]/<code>

6、修改metric-server的deployments配置文件，添加--kubelet-insecure-tls參數設置

metric-server調整部署參數

再次重新部署後無報錯，等待幾分鐘後就有數據上報告metric-server中了，可以通過kubectl top進行驗證測試。

2.4 metric-server api測試

1、安裝完metric-server後會增加一個metrics.k8s.io/v1beta1的API組，該API組通過Aggregator接入apiserver中

metric-server api接口

2、使用命令行查看kubectl top node的監控信息,可以看到CPU和內存的利用率

<code>[root@node-1 1.8+]# kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% node-1 110m 5% 4127Mi 53% node-2 53m 5% 1066Mi 61% node-3 34m 3% 1002Mi 57% /<code>

3、查看pod監控信息,可以看到pod中CPU和內存的使用情況

<code>[root@node-1 1.8+]# kubectl top podsNAME CPU(cores) MEMORY(bytes) haproxy-1-686c67b997-kw8pp 0m 1Mi haproxy-2-689b4f897-7cwmf 0m 1Mi haproxy-ingress-demo-5d487d4fc-5pgjt 0m 1Mi haproxy-ingress-demo-5d487d4fc-pst2q 0m 1Mi haproxy-ingress-demo-5d487d4fc-sr8tm 0m 1Mi ingress-demo-d77bdf4df-7kwbj 0m 1Mi ingress-demo-d77bdf4df-7x6jn 0m 1Mi ingress-demo-d77bdf4df-hr88b 0m 1Mi ingress-demo-d77bdf4df-wc22k 0m 1Mi service-1-7b66bf758f-xj9jh 0m 2Mi service-2-7c7444684d-w9cv9 1m 3Mi /<code>

4、除了用命令行連接metricc-server獲取監控資源，還可以通過API方式鏈接方式獲取，可用API有

如下測試API接口的使用：

<code>a、創建一個kube proxy代理，用於鏈接apiserver，默認將監聽在127的8001端口[root@node-1 ~]# kubectl proxy Starting to serve on 127.0.0.1:8001b、查看node列表的監控數據，可以獲取到所有node的資源監控數據，usage中包含cpu和memory[root@node-1 ~]# curl http://127.0.0.1:8001/apis/metrics.k8s.io/v1beta1/nodes % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 1167 100 1167 0 0 393k 0 --:--:-- --:--:-- --:--:-- 569k{ "kind": "NodeMetricsList", "apiVersion": "metrics.k8s.io/v1beta1", "metadata": { "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes" }, "items": [ { "metadata": { "name": "node-3", "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/node-3", "creationTimestamp": "2019-12-30T14:23:00Z" }, "timestamp": "2019-12-30T14:22:07Z", "window": "30s", "usage": { "cpu": "32868032n", "memory": "1027108Ki" } }, { "metadata": { "name": "node-1", "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/node-1", "creationTimestamp": "2019-12-30T14:23:00Z" }, "timestamp": "2019-12-30T14:22:07Z", "window": "30s", "usage": { "cpu": "108639556n", "memory": "4305356Ki" } }, { "metadata": { "name": "node-2", "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/node-2", "creationTimestamp": "2019-12-30T14:23:00Z" }, "timestamp": "2019-12-30T14:22:12Z", "window": "30s", "usage": { "cpu": "47607386n", "memory": "1119960Ki" } } ]}c、指定某個具體的node訪問到具體node的資源監控數據[root@node-1 ~]# curl http://127.0.0.1:8001/apis/metrics.k8s.io/v1beta1/nodes/node-2{ "kind": "NodeMetrics", "apiVersion": "metrics.k8s.io/v1beta1", "metadata": { "name": "node-2", "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/node-2", "creationTimestamp": "2019-12-30T14:24:39Z" }, "timestamp": "2019-12-30T14:24:12Z", "window": "30s", "usage": { "cpu": "43027609n", "memory": "1120168Ki" }}d、查看所有pod的列表信息curl http://127.0.0.1:8001/apis/metrics.k8s.io/v1beta1/podse、查看某個具體pod的監控數據[root@node-1 ~]# curl http://127.0.0.1:8001/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/haproxy-ingress-demo-5d487d4fc-sr8tm{ "kind": "PodMetrics", "apiVersion": "metrics.k8s.io/v1beta1", "metadata": { "name": "haproxy-ingress-demo-5d487d4fc-sr8tm", "namespace": "default", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/haproxy-ingress-demo-5d487d4fc-sr8tm", "creationTimestamp": "2019-12-30T14:36:30Z" }, "timestamp": "2019-12-30T14:36:13Z", "window": "30s", "containers": [ { "name": "haproxy-ingress-demo", "usage": { "cpu": "0", "memory": "1428Ki" } } ]}/<code>

5、當然也可以通過kubectl -raw的方式訪問接口,如調用node-3的數據

<code>[root@node-1 ~]# kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes/node-3 | jq .{ "kind": "NodeMetrics", "apiVersion": "metrics.k8s.io/v1beta1", "metadata": { "name": "node-3", "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/node-3", "creationTimestamp": "2019-12-30T14:44:46Z" }, "timestamp": "2019-12-30T14:44:09Z", "window": "30s", "usage": { "cpu": "35650151n", "memory": "1026820Ki" }}/<code>

其他近似的接口有：

kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes 獲取所有node的數據

kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes/<node> 獲取特定node數據/<node>

kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods 獲取所有pod的數據

kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/default/pods/haproxy-ingress-demo-5d487d4fc-sr8tm 獲取某個特定pod的數據

3. HPA水平橫向動態擴展

3.1 HPA概述

The Horizontal Pod Autoscaler automatically scales the number of pods in a replication controller, deployment, replica set or stateful set based on observed CPU utilization (or, with custom metrics support, on some other application-provided metrics). Note that Horizontal Pod Autoscaling does not apply to objects that can’t be scaled, for example, DaemonSets.

水平橫向擴展

HPA即Horizontal Pod Autoscaler,Pod水平橫向動態擴展，即根據應用分配資源使用情況，動態增加或者減少Pod副本數量，以實現集群資源的擴容，其實現機制為：

3.2 HPA實現

如下開始延時HPA功能的實現，先創建一個Deployment副本控制器，然後再通過HPA定義資源度量策略，當CPU利用率超過requests分配的80%時即擴容。

1、創建Deployment副本控制器

<code>[root@node-1 ~]# kubectl run hpa-demo --image=nginx:1.7.9 --port=80 --replicas=1 --expose=true --requests="'cpu=200m,memory=64Mi"[root@node-1 ~]# kubectl get deployments hpa-demo -o yamlapiVersion: extensions/v1beta1kind: Deploymentmetadata: annotations: deployment.kubernetes.io/revision: "1" creationTimestamp: "2019-12-31T01:43:24Z" generation: 1 labels: run: hpa-demo name: hpa-demo namespace: default resourceVersion: "14451208" selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/hpa-demo uid: 3b0f29e8-8606-4e52-8f5b-6c960d396136spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: run: hpa-demo strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: creationTimestamp: null labels: run: hpa-demo spec: containers: - image: nginx:1.7.9 imagePullPolicy: IfNotPresent name: hpa-demo ports: - containerPort: 80 protocol: TCP resources: requests: cpu: 200m memory: 64Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30status: availableReplicas: 1 conditions: - lastTransitionTime: "2019-12-31T01:43:25Z" lastUpdateTime: "2019-12-31T01:43:25Z" message: Deployment has minimum availability. reason: MinimumReplicasAvailable status: "True" type: Available - lastTransitionTime: "2019-12-31T01:43:24Z" lastUpdateTime: "2019-12-31T01:43:25Z" message: ReplicaSet "hpa-demo-755bdd875c" has successfully progressed. reason: NewReplicaSetAvailable status: "True" type: Progressing observedGeneration: 1 readyReplicas: 1 replicas: 1 updatedReplicas: 1/<code>

2、創建HPA控制器，基於CPU實現橫向擴展,策略為至少2個Pod，最大5個，targetCPUUtilizationPercentage表示CPU實際使用率佔requests百分比

<code>apiVersion: autoscaling/v1kind: HorizontalPodAutoscalermetadata: name: hpa-demospec: maxReplicas: 5 minReplicas: 2 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: hpa-demo targetCPUUtilizationPercentage: 80/<code>

3、應用HPA規則並查看詳情，由於策略需確保最小2個副本，Deployment默認不是2個副本，因此需要擴容，在詳情日誌中看到副本擴展至2個

<code>[root@node-1 ~]# kubectl apply -f hpa-demo.yaml horizontalpodautoscaler.autoscaling/hpa-demo created#查看HPA列表[root@node-1 ~]# kubectl get horizontalpodautoscalers.autoscaling NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGEhpa-demo Deployment/hpa-demo <unknown>/80% 2 5 0 7s#查看HPA詳情[root@node-1 ~]# kubectl describe horizontalpodautoscalers.autoscaling hpa-demo Name: hpa-demoNamespace: defaultLabels: <none>Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"autoscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"hpa-demo","namespace":"default"},"spe...CreationTimestamp: Tue, 31 Dec 2019 09:52:51 +0800Reference: Deployment/hpa-demoMetrics: ( current / target ) resource cpu on pods (as a percentage of request): <unknown> / 80%Min replicas: 2Max replicas: 5Deployment pods: 1 current / 2 desiredConditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 2Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 1s horizontal-pod-autoscaler New size: 2; reason: Current number of replicas below Spec.MinReplicas #副本擴容至2個，根據MinReplica的策略/<unknown>/<none>/<unknown>/<code>

4、查看Deployment列表校驗確認擴容情況，已達到HPA基礎最小化策略

<code>[root@node-1 ~]# kubectl get deployments hpa-demo --show-labels NAME READY UP-TO-DATE AVAILABLE AGE LABELShpa-demo 2/2 2 2 94m run=hpa-demo[root@node-1 ~]# kubectl get pods -l run=hpa-demoNAME READY STATUS RESTARTS AGEhpa-demo-5fcd9c757d-7q4td 1/1 Running 0 5m10shpa-demo-5fcd9c757d-cq6k6 1/1 Running 0 10m/<code>

5、假如業務增長期間，CPU利用率增高，會自動橫向增加Pod來實現，下面開始通過CPU壓測來演示Deployment的擴展

<code>[root@node-1 ~]# kubectl exec -it hpa-demo-5fcd9c757d-cq6k6 /bin/bashroot@hpa-demo-5fcd9c757d-cq6k6:/# dd if=/dev/zero of=/dev/null 再次查看HPA的日誌，提示已擴容，原因是cpu resource utilization (percentage of request) above target，即CPU資源利用率超過requests設置的百分比[root@node-1 ~]# kubectl describe horizontalpodautoscalers.autoscaling hpa-demo Name: hpa-demoNamespace: defaultLabels: <none>Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"autoscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"hpa-demo","namespace":"default"},"spe...CreationTimestamp: Tue, 31 Dec 2019 09:52:51 +0800Reference: Deployment/hpa-demoMetrics: ( current / target ) resource cpu on pods (as a percentage of request): 99% (199m) / 80%Min replicas: 2Max replicas: 5Deployment pods: 5 current / 5 desiredConditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True ReadyForNewScale recommended size matches current size ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request) ScalingLimited True TooManyReplicas the desired replica count is more than the maximum replica countEvents: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 8m2s horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target查看副本的個數，確認擴容情況，已成功擴容至5個[root@node-1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGEhpa-demo-5fcd9c757d-7q4td 1/1 Running 0 16mhpa-demo-5fcd9c757d-cq6k6 1/1 Running 0 21mhpa-demo-5fcd9c757d-jmb6w 1/1 Running 0 16mhpa-demo-5fcd9c757d-lpxk8 1/1 Running 0 16mhpa-demo-5fcd9c757d-zs6cg 1/1 Running 0 21m/<none>/<code>

6、停止CPU壓測業務，HPA會自定縮減Pod的副本個數，直至滿足條件

<code>[root@node-1 ~]# kubectl describe horizontalpodautoscalers.autoscaling hpa-demoName: hpa-demoNamespace: defaultLabels: <none>Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"autoscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"hpa-demo","namespace":"default"},"spe...CreationTimestamp: Tue, 31 Dec 2019 09:52:51 +0800Reference: Deployment/hpa-demoMetrics: ( current / target ) resource cpu on pods (as a percentage of request): 0% (0) / 80%Min replicas: 2Max replicas: 5Deployment pods: 2 current / 2 desiredConditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True ReadyForNewScale recommended size matches current size ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request) ScalingLimited True TooFewReplicas the desired replica count is increasing faster than the maximum scale rateEvents: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 18m horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target Normal SuccessfulRescale 113s horizontal-pod-autoscaler New size: 2; reason: All metrics below target #縮減至2個pod副本確認副本的個數，已縮減至最小數量2個[root@node-1 ~]# kubectl get pods -l run=hpa-demoNAME READY STATUS RESTARTS AGEhpa-demo-5fcd9c757d-cq6k6 1/1 Running 0 24mhpa-demo-5fcd9c757d-zs6cg 1/1 Running 0 24m/<none>/<code>

通過上面的例子可以知道，HPA可以基於metric-server提供的API監控數據實現水平動態彈性擴展的需求，從而可以根據業務CPU使用情況，動態水平橫向擴展，保障業務的可用性。當前HPA V1擴展使用指標只能基於CPU分配使用率進行擴展，功能相對有限，更豐富的功能需要由HPA V2版來實現，其由不同的API來實現：

參考文獻

資源指標說明：https://kubernetes.io/docs/tasks/debug-application-cluster/resource-metrics-pipeline/

部署官方說明：https://github.com/kubernetes-sigs/metrics-server

(https://github.com/kubernetes-sigs/metrics-server)

當你的才華撐不起你的野心時，你就應該靜下心來學習