Sentinel Slot擴展實踐-流控熔斷預警實現

前言

前幾天公司生產環境一個服務由於流量上升觸發了 Sentinel 的流控機制,然後用戶反饋訪問慢,定位發現是 task 定時任務導致,後面 task 優化之後發佈,流量恢復正常。

這是一個再正常不過的生產問題,可能大部分同學都經歷過,經歷過的大多數是解決問題之後就不了了之,導致事故還有再次發生的可能,最終對用戶造成了不好的體驗。所以我覺得所有的生產問題都需要進行復盤,當然覆盤的目的不是為了追責,而是防止下次再發生同樣的錯誤。那我們就簡單分析一下這個問題,首先肯定是業務層面的疏漏導致 task 發出不合理的大量請求,其二我們的流控只是簡單粗暴的流控,沒有更好的預警措施,導致影響到用戶之後我們才知曉(即流控或熔斷已經觸發)。

那我們的解決方案呢?首先肯定是業務層面的預防,但這不是本文要說的重點,這裡不展開討論了。其次就是預警,就是我們能否在快要觸發流控之前知曉,然後報警到相關負責人提前介入處理,防止觸發流控熔斷。當然也不能完全避免,但是總比流控或熔斷觸發之後在報警要好得多。

由於之前流控用的阿里的 Sentinel,所以本文介紹的具體實現是用 Sentinel 的自定義 slot 功能,這個自定義 slot 卡槽在 Sentinel 官方文檔裡面就一句話帶過,然後加上一個 demo 代碼,我在使用的過程中也遇到過不少坑,所以分享一下結果給大家。

如果大家對 Sentinel 不是很瞭解,可以先去 github 先了解簡單試用一下在閱讀本文。github 地址:https://github.com/alibaba/Sentinel[1]

如果想熟悉自定義 slot 功能建議瞭解一下 Sentinel 的工作原理:https://github.com/alibaba/Sentinel/wiki/Sentinel%E5%B7%A5%E4%BD%9C%E4%B8%BB%E6%B5%81%E7%A8%8B[2]

還有源碼中的 demo 對於自定義 slot 的寫法:https://github.com/alibaba/Sentinel/tree/master/sentinel-demo/sentinel-demo-slot-chain-spi[3]

具體實現

下面介紹下 Sentinel 預警功能的相關實現,使用的前提是你的系統已經在用 Sentinel 的流控或熔斷等功能。

  1. 自定義 CustomSlotChainBuilder 實現 SlotChainBuilder 接口,這裡主要是把我們自定義的 Slot 加到 SlotChain 這個鏈中
<code>import com.alibaba.csp.sentinel.slotchain.ProcessorSlotChain;
import com.alibaba.csp.sentinel.slotchain.SlotChainBuilder;
import com.alibaba.csp.sentinel.slots.DefaultSlotChainBuilder;
import com.qiaofang.tortoise.gateway.component.ApplicationContextUtil;
import com.qiaofang.tortoise.gateway.config.SentinelProperties;
import org.springframework.stereotype.Component;

import javax.annotation.Resource;

/**
* 自定義slot
*
* @author chenhao
*/

public class CustomSlotChainBuilder implements SlotChainBuilder {


@Override
public ProcessorSlotChain build() {
ProcessorSlotChain chain = new DefaultSlotChainBuilder().build();
SentinelProperties sentinelProperties = (SentinelProperties) ApplicationContextUtil.getContext().getBean("sentinelProperties");
chain.addLast(new FlowEarlyWarningSlot(sentinelProperties));
chain.addLast(new DegradeEarlyWarningSlot(sentinelProperties));
return chain;
}
}
/<code>

2.自定義 FlowEarlyWarningSlot、DegradeEarlyWarningSlot 流控熔斷 2 個預警 slot

自定義 FlowEarlyWarningSlot

<code>import com.alibaba.csp.sentinel.context.Context;
import com.alibaba.csp.sentinel.node.DefaultNode;
import com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot;
import com.alibaba.csp.sentinel.slotchain.ResourceWrapper;
import com.alibaba.csp.sentinel.slots.block.flow.FlowRule;
import com.alibaba.csp.sentinel.slots.block.flow.FlowRuleChecker;
import com.alibaba.csp.sentinel.slots.block.flow.FlowRuleManager;
import com.alibaba.csp.sentinel.slots.block.flow.FlowRuleUtil;
import com.alibaba.csp.sentinel.util.AssertUtil;
import com.google.common.collect.Lists;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.BeanUtils;
import org.springframework.util.CollectionUtils;

import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

/**
* 流控預警slot
*
* @author chenhao
*/
public class FlowEarlyWarningSlot2 extends AbstractLinkedProcessorSlot<defaultnode> {

/**
* log
*/

private Logger logger = LoggerFactory.getLogger(this.getClass());

private final FlowRuleChecker checker;

public FlowEarlyWarningSlot2() {
this(new FlowRuleChecker());
}

/**
* Package-private for test.
*
* @param checker flow rule checker
* @since 1.6.1
*/
FlowEarlyWarningSlot2(FlowRuleChecker checker) {
AssertUtil.notNull(checker, "flow checker should not be null");
this.checker = checker;
}


private List<flowrule> getRuleProvider(String resource) {
// Flow rule map should not be null.
List<flowrule> rules = FlowRuleManager.getRules();
List<flowrule> earlyWarningRuleList = Lists.newArrayList();
for (FlowRule rule : rules) {
FlowRule earlyWarningRule = new FlowRule();
BeanUtils.copyProperties(rule, earlyWarningRule);
/**
* 這裡是相當於把規則閾值改成原來的80%,達到提前預警的效果,
* 這裡建議把0.8做成配置
*/
earlyWarningRule.setCount(rule.getCount() * 0.8);
earlyWarningRuleList.add(earlyWarningRule);
}
Map<string>> flowRules = FlowRuleUtil.buildFlowRuleMap(earlyWarningRuleList);
return flowRules.get(resource);
}

/**
* get origin rule
*
* @param resource
* @return
*/
private FlowRule getOriginRule(String resource) {
List<flowrule> originRule = FlowRuleManager.getRules().stream().filter(flowRule -> flowRule.getResource().equals(resource)).collect(Collectors.toList());
if (CollectionUtils.isEmpty(originRule)) {

return null;
}
return originRule.get(0);
}

@Override
public void entry(Context context, ResourceWrapper resourceWrapper, DefaultNode node, int count, boolean prioritized, Object... args)
throws Throwable {
String resource = context.getCurEntry().getResourceWrapper().getName();
List<flowrule> rules = getRuleProvider(resource);
if (rules != null) {
for (FlowRule rule : rules) {
//這裡取到的規則都是配置閾值的80%,這裡如果檢查到閾值了,說明就是到了真實閾值的80%,既可以發報警給對應負責人了
if (!checker.canPassCheck(rule, context, node, count, prioritized)) {
FlowRule originRule = getOriginRule(resource);
String originRuleCount = originRule == null ? "未知" : String.valueOf(originRule.getCount());
logger.info("FlowEarlyWarning:服務{}目前的流量指標已經超過{},接近配置的流控閾值:{},", resource, rule.getCount(), originRuleCount);
//TODO 報警功能自行實現
break;
}
}
}
fireEntry(context, resourceWrapper, node, count, prioritized, args);
}

@Override
public void exit(Context context, ResourceWrapper resourceWrapper, int count, Object... args) {
fireExit(context, resourceWrapper, count, args);
}
}
/<flowrule>/<flowrule>/<string>/<flowrule>/<flowrule>/<flowrule>/<defaultnode>/<code>

DegradeEarlyWarningSlot

<code>import com.alibaba.csp.sentinel.context.Context;
import com.alibaba.csp.sentinel.node.DefaultNode;
import com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot;
import com.alibaba.csp.sentinel.slotchain.ResourceWrapper;
import com.alibaba.csp.sentinel.slots.block.degrade.DegradeRule;
import com.alibaba.csp.sentinel.slots.block.degrade.DegradeRuleManager;

import com.alibaba.csp.sentinel.util.AssertUtil;
import com.google.common.collect.Lists;
import com.qiaofang.tortoise.gateway.config.SentinelProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.BeanUtils;
import org.springframework.util.CollectionUtils;

import java.util.List;
import java.util.stream.Collectors;

/**
* 熔斷預警slot
*
* @author chenhao
*/
public class DegradeEarlyWarningSlot2 extends AbstractLinkedProcessorSlot<defaultnode> {

/**
* log
*/
private Logger logger = LoggerFactory.getLogger(this.getClass());

/**
* 與流控基本一致 就是取原規則的方式不一樣
* @param resource
* @return
*/
private List<degraderule> getRuleProvider(String resource) {
// Flow rule map should not be null.
List<degraderule> rules = DegradeRuleManager.getRules();
List<degraderule> earlyWarningRuleList = Lists.newArrayList();
for (DegradeRule rule : rules) {
DegradeRule earlyWarningRule = new DegradeRule();
BeanUtils.copyProperties(rule, earlyWarningRule);
earlyWarningRule.setCount(rule.getCount() * 0.8);
earlyWarningRuleList.add(earlyWarningRule);
}
return earlyWarningRuleList.stream().filter(rule -> resource.equals(rule.getResource())).collect(Collectors.toList());
}

/**
* get origin rule
*
* @param resource
* @return
*/
private DegradeRule getOriginRule(String resource) {

List<degraderule> originRule = DegradeRuleManager.getRules().stream().filter(rule -> rule.getResource().equals(resource)).collect(Collectors.toList());
if (CollectionUtils.isEmpty(originRule)) {
return null;
}
return originRule.get(0);
}

@Override
public void entry(Context context, ResourceWrapper resourceWrapper, DefaultNode node, int count, boolean prioritized, Object... args)
throws Throwable {
String resource = context.getCurEntry().getResourceWrapper().getName();
List<degraderule> rules = getRuleProvider(resource);
if (rules != null) {
for (DegradeRule rule : rules) {
if (!rule.passCheck(context, node, count)) {
DegradeRule originRule = getOriginRule(resource);
String originRuleCount = originRule == null ? "未知" : String.valueOf(originRule.getCount());
logger.info("DegradeEarlyWarning:服務{}目前的熔斷指標已經超過{},接近配置的熔斷閾值:{},", resource, rule.getCount(), originRuleCount);
break;
}
}
}
fireEntry(context, resourceWrapper, node, count, prioritized, args);
}

@Override
public void exit(Context context, ResourceWrapper resourceWrapper, int count, Object... args) {
fireExit(context, resourceWrapper, count, args);
}
/<degraderule>/<degraderule>/<degraderule>/<degraderule>/<degraderule>/<defaultnode>/<code>

3.在 resources 文件夾下面新增 META-INF.services 文件夾,新增文件 com.alibaba.csp.sentinel.slotchain.SlotChainBuilder(文件名無所謂) 內容如下

<code># 這裡寫你CustomSlotChainBuilder的完整包路徑
com.xxx.sentinel.CustomSlotChainBuilder
/<code>
Sentinel Slot擴展實踐-流控熔斷預警實現

到這裡基本上就可以了,用的過程中還是遇到挺多坑的,簡單列舉幾個吧

  • 直接改 FlowRule 的 count 屬性是不行的,因為底層驗證規則的時候用的是 FlowRule 的 controller 屬性,這個屬性又是私有的,所以直接先拿到原始的配置後通過 FlowRuleUtil 重新生成
  • 調試過程中,DefaultNode 裡面很多方法的值是都是 1s 內有效,從方法 A debug 到方法 B 可能值就沒了,當時一臉懵逼

寫在最後

本人很少寫這種技術博客,所以有什麼問題,或者不嚴謹的地方,大家可以提出來,求輕點噴我哈哈哈

PS:本文是我的一個朋友寫的,大家有好的文章歡迎投稿

[1]

https://github.com/alibaba/Sentinel: https://github.com/alibaba/Sentinel

[2]

https://github.com/alibaba/Sentinel/wiki/Sentinel%E5%B7%A5%E4%BD%9C%E4%B8%BB%E6%B5%81%E7%A8%8B: https://github.com/alibaba/Sentinel/wiki/Sentinel%E5%B7%A5%E4%BD%9C%E4%B8%BB%E6%B5%81%E7%A8%8B

[3]

https://github.com/alibaba/Sentinel/tree/master/sentinel-demo/sentinel-demo-slot-chain-spi: https://github.com/alibaba/Sentinel/tree/master/sentinel-demo/sentinel-demo-slot-chain-spi


Sentinel Slot擴展實踐-流控熔斷預警實現


分享到:


相關文章: