目錄
(1).概述與效果
(2).為何選擇golang開發(附帶不同語言開發的優劣對比)
1.開發語言選型
2.不同開發語言的資源佔用對比
(3).代碼組織結構與文件說明
1.包結構說明
2.不同開發語言的資源佔用對比
(4).如何編譯
1.安裝go包依賴管理工具govendor
2.使用govendor下載包依賴
3.編譯RocketmqExporter
(5).相關編譯文件說明
(6).如何進行容器化部署
1.製作鏡像
2.提供yaml範例
2.1.使用者需要注意&修改的label
2.2.使用者需要注意&修改的環境變量
3.容器化命令
(7).如何進行實體機部署
(8).如何結合prometheus與grafana
1.grafana/prometheus容器化
2.RocketmqExporter容器化
(9).其他相關文章
(1).概述與效果
資源佔用：K8S下，cpu佔用0.01core, 內存佔用10MB。
監控指標：消息堆積數，精確到進程粒度。
監控目的：實時掌控消息消費的健康程度。
數據來源：從rocketmq-console的http請求獲取數據。也就是說RocketmqExporter必須依賴rocketmq-console。好吧，我承認我圖省事兒了^_^。
為什麼自己要重新實現： 官方exporter是java的，相對費資源；另外我們要求對消息堆積數有完備監控，且精確到進程級別。 從topic, consumerGroup, broker,queueId, consumerClientIP, consumerClientPID等維度對消息堆積數進行聚合，如下圖：
效果圖下載地址：
github.com/hepyu/k8s-app-config/blob/master/product/standard/grafana-prometheus-pro/exporter-mq-rocketmq/images/mesage-unconsumed-count.jpg
(2).為何選擇golang開發(附帶不同語言開發的優劣對比)
1.開發語言選型
golang是最適合的選擇。常用選型不外乎java, python, golang。
2.不同開發語言的資源佔用對比
關於鏡像大小與實際資源佔用的生產對比。
特別說明：
java很不適合開發exporter的重要原因有一點就是，"啟動時內存和CPU耗費"與"運行時內存和CPU耗費差異太大"，這就導致容器資源分配時request和max有不小差值， 這個是很不好的，會留下隱患。
rocketmq實例不多還好，但是想象一下如果redis,mysql的exporter也是用java寫，那這個差值就大了，放大到整個集群將成為潛在風險，我想這也是mysql-exporter, redis-exporter官方為什麼用golang寫的原因之一。
但是如果把request和max設置成一樣，又很浪費。
綜上所述，golang是最好的選擇。
(3).代碼組織結構與文件說明
1.包結構說明
2.重點文件說明
(4).如何編譯
有點麻煩，我從開發(IDE用vim)到編譯到image製作都是在linux服務器上，所以都是用的golang體系下原生命令進行操作的。
本工程目錄下提供一個已經編譯好的二進制文件：RocketmqExporter，可以直接使用。
1.安裝go包依賴管理工具govendor
go get -u -v github.com/kardianos/govendor
2.使用govendor下載包依賴
配置環境變量(注意source生效)：export GOPATH=$HOME/go:$HOME/go-workspace
mkdir $HOME/go-workspace/src
然後將本工程clone到目錄$HOME/go-workspace/src。
進入$HOME/go-workspace/src執行govendor命令列出工程依賴：govendor list
<code>m RocketmqExporter/constantm RocketmqExporter/modelm RocketmqExporter/servicem RocketmqExporter/utilsm RocketmqExporter/wrapperm github.com/go-kit/kit/log/levelm github.com/prometheus/client_golang/prometheusm github.com/prometheus/client_golang/prometheus/promhttpm github.com/prometheus/common/promlogm github.com/prometheus/common/promlog/flagm github.com/prometheus/common/versionm gopkg.in/alecthomas/kingpin.v2/<code>
然後執行govendor init,會生成一個vdendor目錄和vendor.json，後邊下載的包依賴都會放到這個目錄下。
<code>vendor.json{"comment": "","ignore": "test","package": [],"rootPath": "hpy-go-rocketmq-exporter"}/<code>下載包依賴到vendor目錄，執行命令：govendor fetch +out，時間比較長(本工程下提供一個已經編譯好的二進制文件：hpy-go-rocketmq-exporter，這個可以直接用於鏡像製作)。
執行完成後，vendor目錄下：
<code>github.comgolang.orggopkg.inRocketmqExportervendor.jsonvendor.json內容：{"comment": "","ignore": "test","package": [{"path": "RocketmqExporter/constant","revision": ""},{"path": "RocketmqExporter/model","revision": ""},{"path": "RocketmqExporter/service","revision": ""},{"path": "RocketmqExporter/utils","revision": ""},{"path": "RocketmqExporter/wrapper","revision": ""},{"checksumSHA1": "MXqUZAuWyiMWV7HC0X2krRinZoI=","path": "github.com/alecthomas/template","revision": "fb15b899a75114aa79cc930e33c46b577cc664b1","revisionTime": "2019-07-18T01:26:54Z"},{"checksumSHA1": "3wt0pTXXeS+S93unwhGoLIyGX/Q=","path": "github.com/alecthomas/template/parse","revision": "fb15b899a75114aa79cc930e33c46b577cc664b1","revisionTime": "2019-07-18T01:26:54Z"},{"checksumSHA1": "VT42paM42J+M52CXStvRwsc1v6g=","path": "github.com/alecthomas/units","revision": "f65c72e2690dc4b403c8bd637baf4611cd4c069b","revisionTime": "2019-09-24T02:57:48Z"},{"checksumSHA1": "0rido7hYHQtfq3UJzVT5LClLAWc=","path": "github.com/beorn7/perks/quantile","revision": "37c8de3658fcb183f997c4e13e8337516ab753e6","revisionTime": "2019-07-31T12:00:54Z"},{"path": "github.com/cespare/xxhash/v2","revision": ""},{"checksumSHA1": "eVc+4p1fDrG3e49wZuztY6D2txA=","path": "github.com/go-kit/kit/log","revision": "9f5354e50d79d79d865f684fe139811cf309870f","revisionTime": "2019-10-18T12:22:45Z"},{"checksumSHA1": "dyVQWAYHLspsCzhDwwfQjvkOtMk=","path": "github.com/go-kit/kit/log/level","revision": "9f5354e50d79d79d865f684fe139811cf309870f","revisionTime": "2019-10-18T12:22:45Z"},{"checksumSHA1": "g8yM1TRZyIjXtopiqbslzgLqtM0=","path": "github.com/go-logfmt/logfmt","revision": "07c9b44f60d7ffdfb7d8efe1ad539965737836dc","revisionTime": "2018-11-22T01:56:15Z"},{"checksumSHA1": "Q3FteGbNvRRUMJqbYbmrcBd2DMo=","path": "github.com/golang/protobuf/proto","revision": "ed6926b37a637426117ccab59282c3839528a700","revisionTime": "2019-10-22T19:55:53Z"},{"checksumSHA1": "abKzFXAn0KDr5U+JON1ZgJ2lUtU=","path": "github.com/kr/logfmt","revision": "b84e30acd515aadc4b783ad4ff83aff3299bdfe0","revisionTime": "2014-02-26T03:06:59Z"},{"checksumSHA1": "bKMZjd2wPw13VwoE7mBeSv5djFA=","path": "github.com/matttproud/golang_protobuf_extensions/pbutil","revision": "c182affec369e30f25d3eb8cd8a478dee585ae7d","revisionTime": "2018-12-31T17:19:20Z"},{"checksumSHA1": "I7hloldMJZTqUx6hbVDp5nk9fZQ=","path": "github.com/pkg/errors","revision": "27936f6d90f9c8e1145f11ed52ffffbfdb9e0af7","revisionTime": "2019-02-27T00:00:51Z"},{"checksumSHA1": "HquvlxEmpILGOdePiJzqL/zMvUY=","path": "github.com/prometheus/client_golang/prometheus","revision": "333f01cef0d61f9ef05ada3d94e00e69c8d5cdda","revisionTime": "2019-10-24T23:19:15Z"},{"checksumSHA1": "UBqhkyjCz47+S19MVTigxJ2VjVQ=","path": "github.com/prometheus/client_golang/prometheus/internal","revision": "333f01cef0d61f9ef05ada3d94e00e69c8d5cdda","revisionTime": "2019-10-24T23:19:15Z"},{"checksumSHA1": "UcahVbxaRZ35Wh58lM9AWEbUEts=","path": "github.com/prometheus/client_golang/prometheus/promhttp","revision": "333f01cef0d61f9ef05ada3d94e00e69c8d5cdda","revisionTime": "2019-10-24T23:19:15Z"},{"checksumSHA1": "V8xkqgmP66sq2ZW4QO5wi9a4oZE=","path": "github.com/prometheus/client_model/go","revision": "14fe0d1b01d4d5fc031dd4bec1823bd3ebbe8016","revisionTime": "2019-08-12T15:41:04Z"},{"checksumSHA1": "vA545Z9FkjGvIHBTAKQOE0nap/k=","path": "github.com/prometheus/common/expfmt","revision": "b5fe7d854c42dc7842e48d1ca58f60feae09d77b","revisionTime": "2019-10-17T12:25:55Z"},{"checksumSHA1": "1Mhfofk+wGZ94M0+Bd98K8imPD4=","path": "github.com/prometheus/common/internal/bitbucket.org/ww/goautoneg","revision": "b5fe7d854c42dc7842e48d1ca58f60feae09d77b","revisionTime": "2019-10-17T12:25:55Z"},{"checksumSHA1": "ccmMs+h9Jo8kE7izqsUkWShD4d0=","path": "github.com/prometheus/common/model","revision": "b5fe7d854c42dc7842e48d1ca58f60feae09d77b","revisionTime": "2019-10-17T12:25:55Z"},{"checksumSHA1": "Pj64Wsr2ji1uTv5l49J89Rff0hY=","path": "github.com/prometheus/common/promlog","revision": "b5fe7d854c42dc7842e48d1ca58f60feae09d77b","revisionTime": "2019-10-17T12:25:55Z"},{"checksumSHA1": "3tSd7cWrq75N2PaoaqAe79Wa+Fw=","path": "github.com/prometheus/common/promlog/flag","revision": "b5fe7d854c42dc7842e48d1ca58f60feae09d77b","revisionTime": "2019-10-17T12:25:55Z"},{"checksumSHA1": "91KYK0SpvkaMJJA2+BcxbVnyRO0=","path": "github.com/prometheus/common/version","revision": "b5fe7d854c42dc7842e48d1ca58f60feae09d77b","revisionTime": "2019-10-17T12:25:55Z"},{"checksumSHA1": "/otbR/D9hWawJC2jDEqxLdYkryk=","path": "github.com/prometheus/procfs","revision": "34c83637414974b5e7d4bd700b49de3c66631989","revisionTime": "2019-10-22T16:02:49Z"},{"checksumSHA1": "ax1TLBC8m/zLs8u//UHHdFf80q4=","path": "github.com/prometheus/procfs/internal/fs","revision": "34c83637414974b5e7d4bd700b49de3c66631989","revisionTime": "2019-10-22T16:02:49Z"},{"checksumSHA1": "sxRjp2SwHqonjR+sHIEXCkfBglI=","path": "github.com/prometheus/procfs/internal/util","revision": "34c83637414974b5e7d4bd700b49de3c66631989","revisionTime": "2019-10-22T16:02:49Z"},{"path": "golang.org/x/sys/windows","revision": ""},{"checksumSHA1": "sToCp8GThnMnsBzsHv+L/tBYQrQ=","path": "gopkg.in/alecthomas/kingpin.v2","revision": "947dcec5ba9c011838740e680966fd7087a71d0d","revisionTime": "2017-12-17T18:08:21Z"}],"rootPath": "hpy-go-rocketmq-exporter"}/<code>此時make會報錯，找不到包github.com/cespare/xxhash/v2，這個是因為prometheus基於依賴於該包，而prometheus是基於gomod構建的，gomod支持能夠識別xxhash後面的v2是指定的版本，常規方法無法下載，可以下載github.com/cespare/xxhash/，然後把該文件夾中的內容都copy到github.com/cespare/xxhash/v2目錄下即可。
在vendor目錄下執行：
git clone github.com/cespare/xxhash.git github.com/cespare/xxhash/v2
3.編譯RocketmqExporter
執行 make 進行編譯，打印信息如下：
<code>>> building binariesGO111MODULE=on /root/go/bin/promu build --prefix /root/go-workspace/src/RocketmqExporter> RocketmqExporter>> running all testsGO111MODULE=on go test -race -mod=vendor ./...? RocketmqExporter [no test files]? RocketmqExporter/constant [no test files]? RocketmqExporter/model [no test files]? RocketmqExporter/service [no test files]? RocketmqExporter/utils [no test files]? RocketmqExporter/wrapper [no test files]>> vetting codeGO111MODULE=on go vet -mod=vendor ./.../<code>
編譯成功後，在目錄下會生成一個二進制文件RocketmqExporter，可以直接執行：./RocketmqExporter，打印如下信息說明成功(不用關心報錯，因為沒有配置參數到環境變量，找不到rocketmq-console)：
level=info ts=2019-11-01T09:19:57.879Z caller=RocketmqExporter.go:27 msg="Starting rocketmq_exporter" version="unsupported value type"
level=info ts=2019-11-01T09:19:57.879Z caller=RocketmqExporter.go:28 msg="Build contenxt" (gogo1.13.3,userroot@future,date111911090-09:17:41)=(MISSING)
level=info ts=2019-11-01T09:19:57.879Z caller=RocketmqExporter.go:34 msg=fmt.metricsPath:
panic: http: invalid pattern
goroutine 1 [running]:
net/http.(*ServeMux).Handle(0xd47080, 0x0, 0x0, 0x9f72c0, 0xc000091ec0)
/usr/local/go/src/net/http/server.go:2397 +0x33a
net/http.Handle(...)
/usr/local/go/src/net/http/server.go:2446
main.main()
/root/go-workspace/src/RocketmqExporter/RocketmqExporter.go:39 +0x720
(5).相關編譯文件說明
Makefile.common中的關鍵代碼：
.PHONY: common-all
#common-all: precheck style check_license lint unused build test
common-all: build test
上述代碼指明瞭構建過程，可以看到我只開啟了build和test兩個構建命令，其餘關閉。
(6).如何進行容器化部署
1.製作鏡像
直接在目錄下執行：rocketmq_exporter.docker-build.sh
鏡像名稱為：hpy253215039/go-rocketmq-exporter，版本為：1.0.0。
鏡像名稱可以自己修改。
2.提供yaml範例
目錄下的範例文件是：go-exporter-deployment-mq-rocketmq-c0.yaml
2.1.使用者需要注意&修改的label
2.2.使用者需要注意&修改的環境變量
3.容器化命令
直接執行: kubectl apply -f go-exporter-deployment-mq-rocketmq-c0.yaml
(7).如何進行實體機部署
將目錄下env.default.config.backup中的內容拷貝到文件~/.bashrc中，然後執行 "source ~/.bashrc"使其生效。
然後運行目錄下的RocketmqExporter二進制文件即可，注意最好使用supervisor進行守護。
golang如果想要獲取自定義變量，必須把自定義變量放到這裡定義：~/.bashrc ，放到/etc/profile中通過os.GetEnv是獲取不到的。
注意環境變量含義，要根據自己的實例情況進行修改：
(8).如何結合prometheus與grafana
筆者提供生產級容器化結合方式。
1.grafana/prometheus容器化
參照工程完成grafana/prometheus的容器化：
github.com/hepyu/k8s-app-config/tree/master/product/standard/grafana-prometheus-pro
上述工程包含消息堆積數的grafana的dashboard。
具體實施步驟和相關生產拓撲描述參見文章：
主要資源位置：
grafana消息堆積數dashboard位於：
github.com/hepyu/k8s-app-config/tree/master/product/standard/grafana-prometheus-pro/grafana/provisioning/dashboards/mq-rocketmq
prometheus.yml配置的抓取規則位於：
github.com/hepyu/k8s-app-config/blob/master/product/standard/grafana-prometheus-pro/prometheus-mq-rocketmq/prometheus-mq-rocketmq-configmap.yaml
2.RocketmqExporter容器化
具體參見文章：
3.使用注意事項
1.rocketmq-dashboard默認只顯示堆積數大於1000的metric項，主要是為了避免顯示太多而凌亂，可以自行修改。
2.當你使用broadcast模式發送消息時，消息堆積數是隻增不減，因為consumer不會提交offset，所以你會看到這些topic的堆積數會很高，在做報警的時候要考慮過濾這類topic。
(9).其他相關文章
閱讀更多 實戰架構 的文章
