我們在使用 Grafana Dashboard 來展示我們的監控圖表的時候，很多時候我們都是去找別人已經做好的 Dashboard 拿過來改一改，但是這樣也造成了很多使用 Grafana 的人員壓根不知道如何去自定義一個 Dashboard，雖然這並不是很困難。這裡我們介紹一個比較新穎（騷）的工具：[DARK](https://github.com/K-Phoen/dark)，全稱 `Dashboards As Resources in Kubernetes.`，意思就是通過 Kubernetes 的資源對象來定義 Grafana Dashboard，實現原理也很簡單，也就是通過 CRD 來定義 Dashboard，然後通過和 Grafana 的 API Token 進行交互實現 Dashboard 的 CRUD。
下面我們來看下如何使用 `DARK` 定義 Grafana Dashboard。首先 Clone 項目代碼：
```shell
$ git clone https://github.com/K-Phoen/dark.git
```
然後安裝 CRD 資源：
```shell
$ kubectl apply -f k8s/crd.yaml
```
然後通過 Secret 對象創建 Grafana 的 API KEYS，在 Grafana 主界面中，選擇左側的配置菜單 -> `API Keys` 創建 API Keys，選擇 `Editor` 的角色：
api keys
創建完成後會彈出一個對話框顯示對應的 `API Keys`，使用這個 KEY 來創建一個對應的 Secret 對象：
```yaml
$ kubectl create secret generic dark-tokens --from-literal=grafana=
```
然後修改 `k8s/cluster-role.yaml` 文件，如下所示：
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: dark
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: dashboards-viewer
rules:
- apiGroups: ["k8s.kevingomez.fr"]
resources: ["grafanadashboards"]
verbs: ["get", "watch", "list"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: dashboards-viewer-cluster
subjects:
- kind: ServiceAccount
name: dark
namespace: default
roleRef:
kind: ClusterRole
name: dashboards-viewer
apiGroup: rbac.authorization.k8s.io
```
然後創建上面的資源對象：
```shell
$ kubectl apply -f k8s/cluster-role.yaml
```
修改 `k8s/deployment.yaml` 文件，將 `GRAFANA_HOST` 環境變量修改成自己的 Grafana 的地址，由於我這裡 Grafana 也安裝在 Kubernetes 集群中的，所以直接用 DNS 形式配置，然後加上上面創建的 `dark` 這個 ServiceAccount：
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: dark
labels:
app: dark
spec:
selector:
matchLabels:
app: dark
template:
metadata:
labels:
app: dark
spec:
volumes:
- name: dark-tokens
secret:
secretName: dark-tokens
serviceAccountName: dark
containers:
- name: dark
image: kphoen/dark:latest
env:
- name: GRAFANA_HOST
value: http://grafana.kube-mon:3000
- name: GRAFANA_TOKEN
valueFrom:
secretKeyRef:
key: grafana
name: dark-tokens
```
修改完成後直接創建上面的 Controller：
```shell
$ kubectl apply -f k8s/deployment.yaml
$ kubectl get pods -l app=dark
NAME READY STATUS RESTARTS AGE
dark-6bd956b8d6-755p2 1/1 Running 0 36m
```
現在 Controller 定義好過後，實際上我們就可以去通過 CRD 對象來定義 Grafana Dashboard 了，如下所示定義了一個 `GrafanaDashboard` 對象，在對象中我們完全就可以根據自己的需求去定義內容了，比如定義 `annotations`、`variables`、`graph`、`table` 都可以，當然最重要的還是數據源要正確，以及查詢語句：（example-dashboards.yaml）
```yaml
apiVersion: k8s.kevingomez.fr/v1
kind: GrafanaDashboard
metadata:
name: example-dashboard
folder: "Test folder"
spec:
title: Awesome dashboard
editable: true
shared_crosshair: true
tags: [generated, yaml]
auto_refresh: 10s
tags_annotations:
- name: Deployments
datasource: "Prometheus"
color: "#5794F2"
tags: ["deploy", "production"]
variables:
- interval:
name: interval
label: Interval
values: ["30s", "1m", "5m", "10m", "30m", "1h", "6h", "12h"]
- query:
name: status
label: HTTP status
datasource: Prometheus
request: "label_values(prometheus_http_requests_total, code)"
- const:
name: percentile
label: Percentile
default: 80
values_map:
50th: "50"
75th: "75"
80th: "80"
85th: "85"
90th: "90"
95th: "95"
99th: "99"
- custom:
name: vX
default: v2
values_map:
v1: v1
v2: v2
rows:
- name: Prometheus
panels:
- graph:
title: HTTP Rate
height: 400px
datasource: Prometheus
targets:
- prometheus:
query: "rate(promhttp_metric_handler_requests_total[$interval])"
legend: "{{handler}} - {{ code }}"
- graph:
title: Heap allocations
height: 400px
datasource: Prometheus
targets:
- prometheus:
query: "go_memstats_heap_alloc_bytes"
legend: "{{job}}"
ref: A
- table:
title: Threads
datasource: Prometheus
targets:
- prometheus:
query: "go_threads"
hidden_columns: ["Time"]
time_series_aggregations:
- label: AVG
type: avg
- label: Current
type: current
- single_stat:
title: Heap Allocations
datasource: Prometheus
targets:
- prometheus:
query: 'go_memstats_heap_alloc_bytes{job="prometheus"}'
unit: bytes
thresholds: ["26000000", "28000000"]
color: ["value"]
- name: "Some text, because it might be useful"
panels:
- text:
title: Some awesome text?
markdown: "Markdown syntax help: [commonmark.org/help](https://commonmark.org/help/)\\n${percentile}"
- text:
title: Some awesome html?
html: "Some awesome html?"
```
同樣直接創建上面的示例文件：
```shell
$ kubectl apply -f example-dashboards.yaml
$ kubectl get dashboards
NAME AGE
example-dashboard 35m
$ kubectl logs -f dark-6bd956b8d6-755p2
W0327 11:10:24.356194 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0327 11:10:24.360886 1 controller.go:87] Setting up event handlers
I0327 11:10:24.362305 1 controller.go:118] Starting dark-controller
I0327 11:10:24.362341 1 controller.go:121] Waiting for informer caches to sync
I0327 11:10:24.462733 1 controller.go:126] Starting workers
I0327 11:10:24.462820 1 controller.go:132] Started workers
I0327 11:13:22.641706 1 controller.go:197] Successfully synced 'default/example-dashboard'
I0327 11:13:22.643061 1 event.go:278] Event(v1.ObjectReference{Kind:"GrafanaDashboard", Namespace:"default", Name:"example-dashboard", UID:"efc6f96f-c7fc-40b5-8b8f-831a95b0a042", APIVersion:"k8s.kevingomez.fr/v1", ResourceVersion:"48490732", FieldPath:""}): type: 'Normal' reason: 'Synced' GrafanaDashboard synced successfully
```
在 Controller 中也可以看到對應的日誌信息，資源對象創建成功以後，現在去 Grafana 頁面上查看可以看到已經新增了一個 `Test folder` 的文件夾以及 `Awesome dashboard`：
查看 Dashboard 就可以看到和上面 CRD 中定義的各種圖表信息了：
grafana dashboard
這樣我們就使用 Kubernetes 資源對象去定義了 Grafana Dashboard 了，這種方式比直接在頁面上去手動配置顯然要更優雅，也符合 `everything as code` 的思想。
閱讀更多 風月笑今生 的文章
