阿里Sentinel框架再探--集群模式

經過之前的Sentinel單機模式，Nacos當配置中心，動態規則配置作為前期研究鋪墊，終於可以開始搭建集群流控。

經過多次的嘗試發現，基於spring-cloud-starter-alibaba-nacos-config 0.1.1.RELEASE版本依賴的sentinel都是1.4.0版本，這個版本的sentinel-web-servlet有些方法有問題不兼容，所以我全部換成1.4.2了，實在是坑。。。

先上pom.xml:

1. <dependency>

2. <groupid>com.alibaba.csp/<groupid>

3. <artifactid>sentinel-core/<artifactid>

4. <version>1.4.2/<version>

5.

6. <dependency>

7. <groupid>com.alibaba.csp/<groupid>

8. <artifactid>sentinel-annotation-aspectj/<artifactid>

9. <version>1.4.2/<version>

10.

11. <dependency>

12. <groupid>com.alibaba.csp/<groupid>

13. <artifactid>sentinel-transport-simple-http/<artifactid>

14. <version>1.4.2/<version>

15.

16. <dependency>

17. <groupid>com.alibaba.csp/<groupid>

18. <artifactid>sentinel-cluster-client-default/<artifactid>

19. <version>1.4.2/<version>

20.

21. <dependency>

22. <groupid>com.alibaba.csp/<groupid>

23. <artifactid>sentinel-cluster-server-default/<artifactid>

24. <version>1.4.2/<version>

25.

26. <dependency>

27. <groupid>com.alibaba.csp/<groupid>

28. <artifactid>sentinel-cluster-common-default/<artifactid>

29. <version>1.4.2/<version>

30.

31. <dependency>

32. <groupid>org.springframework.cloud/<groupid>

33. <artifactid>spring-cloud-alibaba-dependencies/<artifactid>

34. <version>0.1.1.RELEASE/<version>

35. <type>pom/<type>

36. <scope>import/<scope>

37.

38. <dependency>

39. <groupid>org.springframework.cloud/<groupid>

40. <artifactid>spring-cloud-starter-alibaba-sentinel/<artifactid>

41. <version>0.1.1.RELEASE/<version>

42.

43. <dependency>

44. <groupid>org.springframework.cloud/<groupid>

45. <artifactid>spring-cloud-starter-alibaba-nacos-config/<artifactid>

46. <version>0.1.1.RELEASE/<version>

47.

48. <dependency>

49. <groupid>com.alibaba.csp/<groupid>

50. <artifactid>sentinel-datasource-nacos/<artifactid>

51. <version>1.4.2/<version>

52.

53. <dependency>

54. <groupid>com.alibaba.csp/<groupid>

55. <artifactid>sentinel-web-servlet/<artifactid>

56. <version>1.4.2/<version>

57.

Sentinel集群有兩種方式，一種是獨立模式，一種是嵌入模式，具體比較參考官方Sentinel集群流控。

我這裡先實現嵌入模式，測試環境說明：

啟動3臺服務，本機IP是128.160.186.15，不要用127.0.0.1，坑，後面會提到

3個服務分別是9080，9081，9082，對應的Sentinel客戶端api端口是8720，8721，8722，下面配置apigate-cluster-map要用客戶端api端口

我用9080作為嵌入式的Token Server，9081和9082是Token Client。

在Nacos中先寫好配置：

apigate-cluster-map中配置json：

1. [

2. {

3. "clientSet" : ["128.160.186.15@8721", "128.160.186.15@8722"],

4. "ip" : "128.160.186.15",

5. "machineId" : "128.160.186.15@8720",

6. "port" : 18730

7. }

8. ]

巨坑，沒有任何資料強調這裡的port要用默認值18730，我本來以為是9080，配置9080會導致Token Client連接不上Token Server，拿不到token，做不到集群流控。可能是1.4.2的版本問題吧，低版本真的不行。。。

默認值在ClusterServerConfigManager 源碼中我才看到

1. public final class ClusterServerConfigManager {

2.

3. private static boolean embedded = false;

4.

5. /**

6. * Server global transport and scope config.

7. */

8. private static volatile int port = ClusterConstants.DEFAULT_CLUSTER_SERVER_PORT;

apigate-cluster-client-config中配置json：

1. {

2. "requestTimeout" : 60

3. }

apigate-flow-rules中配置json：

1. [

2. {

3. "resource" : "qrband",

4. "controlBehavior": 0,

5. "grade" : 1,

6. "count" : 60,

7. "limitApp" : "default",

8. "strategy" : 0,

9. "clusterMode" : true,

10. "clusterConfig" : {

11. "flowId" : 123,

12. "thresholdType" : 1,

13. "fallbackToLocalWhenFail" : true

14. }

15. }

16. ]

配置列表界面：

按照官方demo，新建SPI服務接口類NacosDataSourceInitFunc：

1. public class NacosDataSourceInitFunc implements InitFunc {

2. private static final String APP_NAME = AppNameUtil.getAppName();

3. private static final String FLOW_POSTFIX = "-flow-rules";

4. // private static final String PARAM_FLOW_POSTFIX = "-param-rules";

5. private static final String SERVER_NAMESPACE_SET_POSTFIX = "-cs-namespace-set";

6. private static final String CLIENT_CONFIG_POSTFIX = "-cc-config";

7. private static final String CLUSTER_MAP_POSTFIX = "-cluster-map";

8.

9. private final String remoteAddress = "localhost";

10. private final String groupId = "DEFAULT_GROUP";

11. private final String flowDataId = APP_NAME + FLOW_POSTFIX;

12. // private final String paramDataId = APP_NAME + PARAM_FLOW_POSTFIX;

13. private final String configDataId = APP_NAME + "-cluster-client-config";

14. private final String clusterMapDataId = APP_NAME + CLUSTER_MAP_POSTFIX;

15. @Override

16. public void init() throws Exception {

17. // Register client dynamic rule data source.

18. initDynamicRuleProperty();

19.

20. // Register token client related data source.

21. // Token client common config:

22. initClientConfigProperty();

23. // Token client assign config (e.g. target token server) retrieved from assign map:

24. initClientServerAssignProperty();

25.

26. // Register token server related data source.

27. // Register dynamic rule data source supplier for token server:

28. registerClusterRuleSupplier();

29. // Token server transport config extracted from assign map:

30. initServerTransportConfigProperty();

31.

32. // Init cluster state property for extracting mode from cluster map data source.

33. initStateProperty();

34. }

35.

36. private void initDynamicRuleProperty() {

37. ReadableDataSource<string>> ruleSource = new NacosDataSource<>(remoteAddress, groupId,/<string>

38. flowDataId, source -> JSON.parseObject(source, new TypeReference<list>>() {}));/<list>

39. FlowRuleManager.register2Property(ruleSource.getProperty());

40.

41. // ReadableDataSource<string>> paramRuleSource = new NacosDataSource<>(remoteAddress, groupId,/<string>

42. // paramDataId, source -> JSON.parseObject(source, new TypeReference<list>>() {}));/<list>

43. // ParamFlowRuleManager.register2Property(paramRuleSource.getProperty());

44. }

45.

46. private void initClientConfigProperty() {

47. ReadableDataSource<string> clientConfigDs = new NacosDataSource<>(remoteAddress, groupId,/<string>

48. configDataId, source -> JSON.parseObject(source, new TypeReference<clusterclientconfig>() {}));/<clusterclientconfig>

49. ClusterClientConfigManager.registerClientConfigProperty(clientConfigDs.getProperty());

50. }

51.

52. private void initServerTransportConfigProperty() {

53. ReadableDataSource<string> serverTransportDs = new NacosDataSource<>(remoteAddress, groupId,/<string>

54. clusterMapDataId, source -> {

55. List<clustergroupentity> groupList = JSON.parseObject(source, new TypeReference<list>>() {});/<list>/<clustergroupentity>

56. return Optional.ofNullable(groupList)

57. .flatMap(this::extractServerTransportConfig)

58. .orElse(null);

59. });

60. ClusterServerConfigManager.registerServerTransportProperty(serverTransportDs.getProperty());

61. }

62.

63. private void registerClusterRuleSupplier() {

64. // Register cluster flow rule property supplier which creates data source by namespace.

65. // Flow rule dataId format: ${namespace}-flow-rules

66. ClusterFlowRuleManager.setPropertySupplier(namespace -> {

67. ReadableDataSource<string>> ds = new NacosDataSource<>(remoteAddress, groupId,/<string>

68. namespace + FLOW_POSTFIX, source -> JSON.parseObject(source, new TypeReference<list>>() {}));/<list>

69. return ds.getProperty();

70. });

71. // Register cluster parameter flow rule property supplier which creates data source by namespace.

72. // ClusterParamFlowRuleManager.setPropertySupplier(namespace -> {

73. // ReadableDataSource<string>> ds = new NacosDataSource<>(remoteAddress, groupId,/<string>

74. // namespace + PARAM_FLOW_POSTFIX, source -> JSON.parseObject(source, new TypeReference<list>>() {}));/<list>

75. // return ds.getProperty();

76. // });

77. }

78.

79. private void initClientServerAssignProperty() {

80. // Cluster map format:

81. // [{"clientSet":["112.12.88.66@8729","112.12.88.67@8727"],"ip":"112.12.88.68","machineId":"112.12.88.68@8728","port":11111}]

82. // machineId: , commandPort for port exposed to Sentinel dashboard (transport module)

83. ReadableDataSource<string> clientAssignDs = new NacosDataSource<>(remoteAddress, groupId,/<string>

84. clusterMapDataId, source -> {

85. List<clustergroupentity> groupList = JSON.parseObject(source, new TypeReference<list>>() {});/<list>/<clustergroupentity>

86. return Optional.ofNullable(groupList)

87. .flatMap(this::extractClientAssignment)

88. .orElse(null);

89. });

90. ClusterClientConfigManager.registerServerAssignProperty(clientAssignDs.getProperty());

91. }

92.

93. private void initStateProperty() {

94. // Cluster map format:

95. // [{"clientSet":["112.12.88.66@8729","112.12.88.67@8727"],"ip":"112.12.88.68","machineId":"112.12.88.68@8728","port":11111}]

96. // machineId: , commandPort for port exposed to Sentinel dashboard (transport module)

97. ReadableDataSource<string> clusterModeDs = new NacosDataSource<>(remoteAddress, groupId,/<string>

98. clusterMapDataId, source -> {

99. List<clustergroupentity> groupList = JSON.parseObject(source, new TypeReference<list>>() {});/<list>/<clustergroupentity>

100. return Optional.ofNullable(groupList)

101. .map(this::extractMode)

102. .orElse(ClusterStateManager.CLUSTER_NOT_STARTED);

103. });

104. ClusterStateManager.registerProperty(clusterModeDs.getProperty());

105. }

106.

107. private int extractMode(List<clustergroupentity> groupList) {/<clustergroupentity>

108. // If any server group machineId matches current, then it's token server.

109. if (groupList.stream().anyMatch(this::machineEqual)) {

110. return ClusterStateManager.CLUSTER_SERVER;

111. }

112. // If current machine belongs to any of the token server group, then it's token client.

113. // Otherwise it's unassigned, should be set to NOT_STARTED.

114. boolean canBeClient = groupList.stream()

115. .flatMap(e -> e.getClientSet().stream())

116. .filter(Objects::nonNull)

117. .anyMatch(e -> e.equals(getCurrentMachineId()));

118. return canBeClient ? ClusterStateManager.CLUSTER_CLIENT : ClusterStateManager.CLUSTER_NOT_STARTED;

119. }

120.

121. private Optional<servertransportconfig> extractServerTransportConfig(List<clustergroupentity> groupList) {/<clustergroupentity>/<servertransportconfig>

122. return groupList.stream()

123. .filter(this::machineEqual)

124. .findAny()

125. .map(e -> new ServerTransportConfig().setPort(e.getPort()).setIdleSeconds(600));

126. }

127.

128. private Optional<clusterclientassignconfig> extractClientAssignment(List<clustergroupentity> groupList) {/<clustergroupentity>/<clusterclientassignconfig>

129. if (groupList.stream().anyMatch(this::machineEqual)) {

130. return Optional.empty();

131. }

132. // Build client assign config from the client set of target server group.

133. for (ClusterGroupEntity group : groupList) {

134. if (group.getClientSet().contains(getCurrentMachineId())) {

135. String ip = group.getIp();

136. Integer port = group.getPort();

137. return Optional.of(new ClusterClientAssignConfig(ip, port));

138. }

139. }

140. return Optional.empty();

141. }

142.

143. private boolean machineEqual(/*@Valid*/ ClusterGroupEntity group) {

144. return getCurrentMachineId().equals(group.getMachineId());

145. }

146.

147. private String getCurrentMachineId() {

148. // Note: this may not work well for container-based env.

149. return HostNameUtil.getIp() + SEPARATOR + TransportConfig.getRuntimePort();

150. }

151.

152. private static final String SEPARATOR = "@";

153. }

HostNameUtil.getIp() + SEPARATOR + TransportConfig.getRuntimePort()

注意這裡得到的是本機實際IP，而不是127.0.0.1，端口得到的是-Dcsp.sentinel.api.port指定的客戶端api端口

與apigate-cluster-map配置相對應的建個實體類ClusterGroupEntity：

1. public class ClusterGroupEntity {

2. private String machineId;

3. private String ip;

4. private Integer port;

5.

6. private Set<string> clientSet;/<string>

7.

8. public String getMachineId() {

9. return machineId;

10. }

11.

12. public ClusterGroupEntity setMachineId(String machineId) {

13. this.machineId = machineId;

14. return this;

15. }

16.

17. public String getIp() {

18. return ip;

19. }

20.

21. public ClusterGroupEntity setIp(String ip) {

22. this.ip = ip;

23. return this;

24. }

25.

26. public Integer getPort() {

27. return port;

28. }

29.

30. public ClusterGroupEntity setPort(Integer port) {

31. this.port = port;

32. return this;

33. }

34.

35. public Set<string> getClientSet() {/<string>

36. return clientSet;

37. }

38.

39. public ClusterGroupEntity setClientSet(Set<string> clientSet) {/<string>

40. this.clientSet = clientSet;

41. return this;

42. }

43.

44. @Override

45. public String toString() {

46. return "ClusterGroupEntity{" +

47. "machineId='" + machineId + '\\'' +

48. ", ip='" + ip + '\\'' +

49. ", port=" + port +

50. ", clientSet=" + clientSet +

51. '}';

52. }

53. }

依次啟動9080，9081，9082這3臺服務，注意每次啟動時要將Nacos中配置文件的server.port修改下，還要修改idea啟動項VM參數中-Dproject.name=apigate -Dcsp.sentinel.log.use.pid=true -Dcsp.sentinel.api.port=8720的端口。集群模式下-Dcsp.sentinel.log.use.pid=true必須要有。

dashboard中的配置全是從Nacos同步過去的。

機器列表：

集群流控TokenServer列表：

這裡最大允許QPS的30000是指Token Client對Token Server請求token的QPS，是對Server的保護，默認值是在源碼ServerFlowConfig.DEFAULT_MAX_ALLOWED_QPS 中指定的。

1. public class ServerFlowConfig {

2.

3. public static final double DEFAULT_EXCEED_COUNT = 1.0d;

4. public static final double DEFAULT_MAX_OCCUPY_RATIO = 1.0d;

5.

6. public static final int DEFAULT_INTERVAL_MS = 1000;

7. public static final int DEFAULT_SAMPLE_COUNT= 10;

8. public static final double DEFAULT_MAX_ALLOWED_QPS= 30000;

集群流控TokenServer編輯：

集群流控TokenClient列表：

集群流控TokenServer連接詳情：

集群流控規則：

為了做集群請求測試，搭建Nginx做負載均衡轉發3臺服務，nginx.conf增加配置：

1. http {

2. include mime.types;

3. default_type application/octet-stream;

4.

5. #log_format main '$remote_addr - $remote_user [$time_local] "$request" '

6. # '$status $body_bytes_sent "$http_referer" '

7. # '"$http_user_agent" "$http_x_forwarded_for"';

8.

9. #access_log logs/access.log main;

10.

11. sendfile on;

12. #tcp_nopush on;

13.

14. #keepalive_timeout 0;

15. keepalive_timeout 65;

16.

17. #gzip on;

18.

19. #服務器的集群

20. upstream apigate { #服務器集群名字

21. server 127.0.0.1:9080 weight=1;#服務器配置 weight是權重的意思，權重越大，分配的概率越大。

22. server 127.0.0.1:9081 weight=1;

23. server 127.0.0.1:9082 weight=1;

24. }

25.

26. server {

27. listen 88;

28. server_name localhost;

29.

30. #charset koi8-r;

31.

32. #access_log logs/host.access.log main;

33.

34. location / {

35. proxy_pass http://apigate;

36. }

用JMeter測試請求http://localhost:88/api/qrband/123456789，

在dashboard中觀察實時監控，確實流控在60QPS，似乎有一點突破閾值。

同樣，在Nacos中修改流控count，dashboard中實時監控中可以看到是實時生效的。

動態切換Token Server和Client請求：http://localhost:8720/setClusterMode?mode=0

http://localhost:8721/setClusterMode?mode=1
將9080服務改為Token Client，將9081改為Token Server

觀察dashboard中的集群Token Server列表發現9082會自動切換Token Server為9081，而9080需要手動添加Token Server的IP128.160.186.15即可連上Token Server。

Sentinel資料少，我的Spring Cloud版本低，這一路走來不容易。。。

不過Sentinel本身還是很強大的

閱讀更多 銀行小馬農的日常 的文章

關鍵字: public return sentinel