elasticsearch缃戝叧鑺傜偣 (elasticsearch闆嗙兢鏁版嵁瀛樺偍杩囩▼)

目标

  1. 监控接口效率
  2. 监控接口成功率
  3. 分析热点服务或热点接口

效果展示

在kibana中查看请求的信息:

elasticsearch闆嗙兢鏁版嵁瀛樺偍杩囩▼,elasticsearch缃戝叧鑺傜偣

通过kibana的图表功能展示

elasticsearch闆嗙兢鏁版嵁瀛樺偍杩囩▼,elasticsearch缃戝叧鑺傜偣

代码实现

通过两个过滤器来做请求信息的记录,通过网关流水号在redis中存储一个hash的数据结构将我们需要的信息保存起来。在网关处理完成后将流水号插入到redis的一个队列中,由另外一个服务(task)监控redis队列并将hash数据存储到es中。

需要注意:

  1. 在redis中的hash数据需要设置超时时间防止task服务停止导致redis内存耗尽
  2. 需要通过es索引模版来生成索引,可以减少es磁盘使用率
  3. es索引按天创建
  4. 将请求中的网关流水号设置为es文档id

配置文件:

platform-cloud/platform-cloud-web/platform-cloud-web-gateway/src/main/resources/application.yml

elasticsearch闆嗙兢鏁版嵁瀛樺偍杩囩▼,elasticsearch缃戝叧鑺傜偣

CacheParams将请求参数和请求内容存入redis中。

类名:CacheParamsGatewayFilterFactory

文件路径:

platform-cloud/platform-cloud-web/platform-cloud-web-gateway/src/main/java/com/tinem/platform/web/gateway/filter/CacheParamsGatewayFilterFactory.java
/*
 * Copyright 2013-2019 the original author or authors.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      https://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package com.tinem.platform.web.gateway.filter;

import cn.hutool.core.util.StrUtil;
import com.google.common.net.HttpHeaders;
import com.tinem.platform.module.pojo.co.GatewayHeadName;
import com.tinem.platform.module.pojo.co.RedisKeyEnum;
import com.tinem.platform.web.gateway.util.GatewayUtil;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.cloud.gateway.filter.GatewayFilter;
import org.springframework.cloud.gateway.filter.factory.AbstractGatewayFilterFactory;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;

import javax.annotation.Resource;
import java.util.Hashtable;
import java.util.Map;
import java.util.concurrent.TimeUnit;

/**
 * GatewayFilter that modifies the respons body.
 */
@Slf4j
@Component
public class CacheParamsGatewayFilterFactory extends
        AbstractGatewayFilterFactory<Object> {

    @Value("${spring.profiles.active}")
    String active;
    @Resource
    StringRedisTemplate stringRedisTemplate;

    @Override
    public GatewayFilter apply(Object config) {
        return (exchange, chain) -> {
            String gatewayRequestId = GatewayUtil.getReq(exchange);

            Map<String,String> req = new Hashtable<>(30,1);

            // --------浏览器信息 ------------
            req.put(HttpHeaders.USER_AGENT, StrUtil.emptyToDefault(exchange.getRequest().getHeaders().getFirst(HttpHeaders.USER_AGENT),""));
            req.put(HttpHeaders.ORIGIN,StrUtil.emptyToDefault(exchange.getRequest().getHeaders().getFirst(HttpHeaders.ORIGIN),""));
            req.put(HttpHeaders.REFERER,StrUtil.emptyToDefault(exchange.getRequest().getHeaders().getFirst(HttpHeaders.REFERER),""));
            req.put(HttpHeaders.HOST,StrUtil.emptyToDefault(exchange.getRequest().getHeaders().getFirst(HttpHeaders.HOST),""));

            // ---------cloudflare ------
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_IP,StrUtil.emptyToDefault(exchange.getRequest().getHeaders().getFirst("Cf-Connecting-Ip"),""));
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_IP_COUNTRY,StrUtil.emptyToDefault(exchange.getRequest().getHeaders().getFirst("Cf-Ipcountry"),""));

            // ---------请求参数缓存 ------
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_TIMESTAMP, String.valueOf(GatewayUtil.getReqTimestamp(exchange)));
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_SERVICE,GatewayUtil.getApiService(exchange));
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_METHOD,GatewayUtil.getApiMethod(exchange));
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_VERSION,StrUtil.emptyToDefault(exchange.getRequest().getHeaders().getFirst(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_VERSION),""));
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_API_REQUEST_ID,GatewayUtil.getApiReq(exchange));
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_JWT,GatewayUtil.getJwt(exchange));
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_CHARSET,StrUtil.emptyToDefault(exchange.getRequest().getHeaders().getFirst(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_CHARSET),""));
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_LANG,StrUtil.emptyToDefault(exchange.getRequest().getHeaders().getFirst(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_LANG),""));

            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_SIGN_TYPE, StrUtil.emptyToDefault(exchange.getRequest().getHeaders().getFirst(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_SIGN_TYPE),""));
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_SIGN, StrUtil.emptyToDefault(exchange.getRequest().getHeaders().getFirst(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_SIGN),""));

            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_CRYPTO_TYPE,StrUtil.emptyToDefault(exchange.getRequest().getHeaders().getFirst(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_CRYPTO_TYPE),""));
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_CRYPTO_KEY,StrUtil.emptyToDefault(exchange.getRequest().getHeaders().getFirst(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_CRYPTO_KEY),""));
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_CRYPTO_IV,StrUtil.emptyToDefault(exchange.getRequest().getHeaders().getFirst(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_CRYPTO_IV),""));

            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_DEBUG,StrUtil.emptyToDefault(exchange.getRequest().getHeaders().getFirst(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_DEBUG),""));
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_DEVICE,StrUtil.emptyToDefault(exchange.getRequest().getHeaders().getFirst(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_DEVICE),""));

            // ---------网关参数缓存 ------
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQUEST_ID,gatewayRequestId);
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_CONTEXT_ACCEPT_TIMESTAMP,String.valueOf(GatewayUtil.getReqAcceptTimestamp(exchange)));
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_CONTEXT_JTI,GatewayUtil.getJit(exchange));
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_CONTEXT_CLIENT_ID,GatewayUtil.getClient(exchange));
            req.put(GatewayHeadName.X_PLATFORM_GATEWAY_CONTEXT_ENV,active);

            String userId = exchange.getAttribute(GatewayHeadName.X_PLATFORM_GATEWAY_CONTEXT_USER_ID);
            if(StrUtil.isNotEmpty(userId)){
                req.put(GatewayHeadName.X_PLATFORM_GATEWAY_CONTEXT_USER_ID,userId);
            }
            // ---------请求内容 ------
            String requestBody = exchange.getAttribute(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_BODY);
            if(StrUtil.isNotBlank(requestBody)){
                req.put(GatewayHeadName.X_PLATFORM_GATEWAY_REQ_BODY,requestBody);
            }

            // ---------将数据保存到redis ------
            String key = RedisKeyEnum.gateway_req_info.getKey(gatewayRequestId);
            stringRedisTemplate.opsForHash().putAll(key,req);
            // ---------设置10分钟后过期 ------
            stringRedisTemplate.expire(key,10, TimeUnit.MINUTES);
            return chain.filter(exchange);
        };
    }
}

CacheResponse将请求返回数据和请求的状态码状态信息存储到redis中。

类名:CacheResponseGatewayFilterFactory

文件路径:

platform-cloud/platform-cloud-web/platform-cloud-web-gateway/src/main/java/com/tinem/platform/web/gateway/filter/CacheResponseGatewayFilterFactory.java
/*
 * Copyright 2013-2019 the original author or authors.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      https://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package com.tinem.platform.web.gateway.filter;

import cn.hutool.core.util.StrUtil;
import com.alibaba.fastjson.JSON;
import com.tinem.platform.module.pojo.co.GatewayHeadName;
import com.tinem.platform.module.pojo.co.RedisKeyEnum;
import com.tinem.platform.web.gateway.util.GatewayUtil;
import com.tinem.platform.web.gateway.util.MDCUtil;
import lombok.extern.slf4j.Slf4j;
import org.reactivestreams.Publisher;
import org.springframework.cloud.gateway.filter.GatewayFilter;
import org.springframework.cloud.gateway.filter.GatewayFilterChain;
import org.springframework.cloud.gateway.filter.NettyWriteResponseFilter;
import org.springframework.cloud.gateway.filter.factory.AbstractGatewayFilterFactory;
import org.springframework.core.Ordered;
import org.springframework.core.io.buffer.DataBuffer;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.http.server.reactive.ServerHttpResponse;
import org.springframework.http.server.reactive.ServerHttpResponseDecorator;
import org.springframework.stereotype.Component;
import org.springframework.web.server.ServerWebExchange;
import reactor.core.publisher.Mono;

import javax.annotation.Resource;
import java.util.Hashtable;
import java.util.Map;
import java.util.concurrent.TimeUnit;

/**
 * GatewayFilter that modifies the respons body.
 */
@Slf4j
@Component
public class CacheResponseGatewayFilterFactory extends
        AbstractGatewayFilterFactory<Object> {
    @Resource
    StringRedisTemplate stringRedisTemplate;

    @Override
    public GatewayFilter apply(Object config) {
        ModifyResponseGatewayFilter gatewayFilter = new ModifyResponseGatewayFilter();
        return gatewayFilter;
    }
    public class ModifyResponseGatewayFilter implements GatewayFilter, Ordered {
        @Override
        public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
            return chain.filter(exchange.mutate().response(decorate(exchange)).build());
        }

        @SuppressWarnings("unchecked")
        ServerHttpResponse decorate(ServerWebExchange exchange) {
            return new ServerHttpResponseDecorator(exchange.getResponse()) {

                @Override
                public Mono<Void> writeWith(Publisher<? extends DataBuffer> body) {
                    this.cacheResponse(exchange);
                    return super.writeWith(body);
                }

                @Override
                public Mono<Void> writeAndFlushWith(
                        Publisher<? extends Publisher<? extends DataBuffer>> body) {
                    this.cacheResponse(exchange);
                    return super.writeAndFlushWith(body);
                }
                private void cacheResponse(ServerWebExchange exchange){

                    MDCUtil.set(exchange);

                    String resTimestamp = exchange.getResponse().getHeaders().getFirst(GatewayHeadName.X_PLATFORM_GATEWAY_RES_TIMESTAMP);
                    String gatewayResCode = exchange.getResponse().getHeaders().getFirst(GatewayHeadName.X_PLATFORM_GATEWAY_RES_CODE);
                    String gatewayResSuccess = exchange.getResponse().getHeaders().getFirst(GatewayHeadName.X_PLATFORM_GATEWAY_RES_SUCCES);
                    String gatewayResMessage = exchange.getResponse().getHeaders().getFirst(GatewayHeadName.X_PLATFORM_GATEWAY_RES_MESSAGE);

                    Map<String,String> req = new Hashtable<>(7,1);
                    req.put(GatewayHeadName.X_PLATFORM_GATEWAY_RES_TIMESTAMP,resTimestamp);
                    req.put(GatewayHeadName.X_PLATFORM_GATEWAY_RES_CODE,gatewayResCode);
                    req.put(GatewayHeadName.X_PLATFORM_GATEWAY_RES_SUCCES,gatewayResSuccess);
                    req.put(GatewayHeadName.X_PLATFORM_GATEWAY_RES_MESSAGE,gatewayResMessage);

                    String resCode = exchange.getResponse().getHeaders().getFirst(GatewayHeadName.X_PLATFORM_RES_CODE);
                    String resSuccess = exchange.getResponse().getHeaders().getFirst(GatewayHeadName.X_PLATFORM_RES_SUCCESS);
                    String resMessage = exchange.getResponse().getHeaders().getFirst(GatewayHeadName.X_PLATFORM_RES_MESSAGE);
                    req.put(GatewayHeadName.X_PLATFORM_RES_CODE, StrUtil.nullToDefault(resCode,""));
                    req.put(GatewayHeadName.X_PLATFORM_RES_SUCCESS,StrUtil.nullToDefault(resSuccess,""));
                    req.put(GatewayHeadName.X_PLATFORM_RES_MESSAGE,StrUtil.nullToDefault(resMessage,""));

                    String gatewayRequestId = GatewayUtil.getReq(exchange);
                    String key = RedisKeyEnum.gateway_req_info.getKey(gatewayRequestId);
                    try {
                        stringRedisTemplate.opsForHash().putAll(key,req);
                    }catch (Exception e){
                        log.error("redis cache error:{},{}", key,JSON.toJSONString(req));
                        log.error("",e);
                    }
                    // ---------设置5分钟后过期 ------
                    stringRedisTemplate.expire(key,5, TimeUnit.MINUTES);
                    // ---------加入redis队列  ------
                    stringRedisTemplate.opsForList().rightPush(RedisKeyEnum.gateway_req_info_query.getKey(),gatewayRequestId);
                    log.info("request info to redis query.");
                }
            };
        }

        @Override
        public int getOrder() {
            return NettyWriteResponseFilter.WRITE_RESPONSE_FILTER_ORDER - 2;
        }

    }
}

将Redis中数据同步到ES

类:GatewayReqSaveToElasticsearch

文件路径

platform-cloud/platform-cloud-service/platform-cloud-service-task/src/main/java/com/tinem/platform/service/task/consumer/redis/GatewayReqSaveToElasticsearch.java
package com.tinem.platform.service.task.consumer.redis;

import cn.hutool.core.collection.CollUtil;
import cn.hutool.core.date.DateUtil;
import cn.hutool.core.util.StrUtil;
import cn.hutool.http.HttpUtil;
import com.alibaba.fastjson.JSONObject;
import com.tinem.platform.module.pojo.co.GatewayHeadName;
import com.tinem.platform.module.pojo.co.RedisKeyEnum;
import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Configurable;
import org.springframework.boot.ApplicationArguments;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.dao.InvalidDataAccessApiUsageException;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;
import redis.clients.jedis.exceptions.JedisException;

import javax.annotation.Resource;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
import java.util.NoSuchElementException;
import java.util.concurrent.TimeUnit;

/**
 * 将网关请求信息保存到ES中
 * //TODO 可以优化成批量获取和批量插入
 */
@Slf4j
@Data
@Component
@Configurable
@ConfigurationProperties(prefix = "gateway.req.save.elasticsearch")
public class GatewayReqSaveToElasticsearch implements ApplicationRunner {

    // 执行线程数
    private int threadNum;
    // es地址
    private String elasticsearchUrl;

    @Resource
    StringRedisTemplate stringRedisTemplate;

    @Override
    public void run(ApplicationArguments args) throws Exception {
        for (int i = 0; i < threadNum; i++) {
            Thread thread = new Thread(()->{
                while (true){
                    try {
                        // 从redis队列中获取请求流水,以阻塞队列方式,如果没有流水则最长等待1天
                        String gatewayRequestId = stringRedisTemplate.opsForList().leftPop(RedisKeyEnum.gateway_req_info_query.getKey(),1, TimeUnit.DAYS);
                        // 如果没有获取到流水号
                        if(StrUtil.isBlank(gatewayRequestId)){
                            continue;
                        }
                        String redisGatewayRequestId = RedisKeyEnum.gateway_req_info.getKey(gatewayRequestId);
                        // 从redis中获取请求信息
                        Map<Object, Object> req = stringRedisTemplate.opsForHash().entries(redisGatewayRequestId);
                        if(CollUtil.isEmpty(req)){
                            continue;
                        }
                        // 将请求信息保存到ES
                        String s = HttpUtil.post(StrUtil.format(elasticsearchUrl,new HashMap(){{
                            String millis = (String) req.getOrDefault(GatewayHeadName.X_PLATFORM_GATEWAY_CONTEXT_ACCEPT_TIMESTAMP,System.currentTimeMillis()+"");
                            put("date", DateUtil.formatDate(new Date(Long.parseLong(millis))));
                            put("env", req.get(GatewayHeadName.X_PLATFORM_GATEWAY_CONTEXT_ENV));
                            put(GatewayHeadName.X_PLATFORM_GATEWAY_REQUEST_ID, req.get(GatewayHeadName.X_PLATFORM_GATEWAY_REQUEST_ID));
                        }}),JSONObject.toJSONString(req));
                        // 清理redis中的数据
                        stringRedisTemplate.delete(redisGatewayRequestId);
                        log.debug(s);
                    }catch (InvalidDataAccessApiUsageException e){
                        if(e.getCause() instanceof JedisException && e.getCause().getCause() instanceof NoSuchElementException){
                            //如果redis中没有数据,则睡眠60秒
                            try {
                                Thread.sleep(60*1000);
                            } catch (InterruptedException ex) {
                                ex.printStackTrace();
                            }
                        }else{
                            log.error("",e);
                        }
                    }catch (Exception e){
                        log.error("",e);
                    }

                }
            });
            thread.setName("gatewayReqSaveElasticsearch-"+i);
            thread.start();
        }

    }
}

在ES中建立索引模版

请求:

地址:/_template/data-tinem-all-gateway-request
方法:put
{
    "index_patterns": ["data-tinem-*-gateway-request-*"],                   // 可以通过"book_*"和"bar*"来适配, template字段已过期
    "order": 0,                                             // 模板的权重, 多个模板的时候优先匹配用, 值越大, 权重越高
    "aliases": {
        "data-tinem-all-gateway-request": {}                                          // 索引对应的别名
    },
    "settings": {
        "number_of_shards": 1,                              // 索引分片数量
        "number_of_replicas": 1                             // 副本数量
    }, 
    "mappings": {
        "_source": {
            "enabled": true                                // 如果我们只关心查询的评分结果, 而不用查看原始文档的内容, 就设置"_source{"enabled": false}.—— 这能节省磁盘空间并减少磁盘IO上的开销.
        }, 
        "dynamic": "true",                               // 只用定义的字段, 关闭默认的自动类型推断
        "properties": {
            // 收到请求的时间
            "x-platform-gateway-accept-timestamp": {
                "type": "date"
            },
            // 客户端id
            "x-platform-gateway-context-client_id": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 用户id
            "x-platform-gateway-context-user_id": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 网关请求流水号
            "x-platform-gateway-req-api_request_id": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 请求报文
            "x-platform-gateway-req-body": {
                "type": "text"
            },
            // 字符编码
            "x-platform-gateway-req-charset": {
                "type": "keyword",
                "ignore_above": 32
            },
            // 加密算法:IV
            "x-platform-gateway-req-crypto_iv": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 加密算法:KEY
            "x-platform-gateway-req-crypto_key": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 加密算法
            "x-platform-gateway-req-crypto_type": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 是否开启debug模式
            "x-platform-gateway-req-debug": {
                "type": "keyword",
                "ignore_above": 256
            },
            // jwt
            "x-platform-gateway-req-jwt": {
                "type": "keyword",
                "ignore_above": 4096
            },
            // jwt jit
            "x-platform-gateway-context-jti": {
                "type": "keyword",
                "ignore_above": 64
            },
            // 请求ip
            "x-platform-gateway-req-ip": {
                "type": "keyword",
                "ignore_above": 64
            },
            // 请求IP归属地
            "x-platform-gateway-req-ip-country": {
                "type": "keyword",
                "ignore_above": 64
            },
            // 设备ID
            "x-platform-gateway-req-device": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 浏览器信息
            "User-Agent": {
                "type": "keyword",
                "ignore_above": 1024
            },
            // 请求来自于哪个站点
            "Origin": {
                "type": "keyword",
                "ignore_above": 1024
            },
            // 该网页是从哪个页面链接过来的
            "Referer": {
                "type": "keyword",
                "ignore_above": 1024
            },
            // 域名
            "Host": {
                "type": "keyword",
                "ignore_above": 1024
            },
            // 语言
            "x-platform-gateway-req-lang": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 接口
            "x-platform-gateway-req-method": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 服务
            "x-platform-gateway-req-service": {
                "type": "keyword",
                "ignore_above": 512
            },
            // 签名
            "x-platform-gateway-req-sign": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 签名算法
            "x-platform-gateway-req-sign_type": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 客户端发起请求的时间
            "x-platform-gateway-req-timestamp": {
                "type": "date"
            },
            // 接口版本
            "x-platform-gateway-req-version": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 客户端请求流水号
            "x-platform-gateway-request_id": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 环境
            "x-platform-gateway-request-env": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 返回数据
            "x-platform-gateway-res-body": {
                "type": "text"
            },
            // 返回状态码
            "x-platform-gateway-res-code": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 返回消息
            "x-platform-gateway-res-message": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 返回是否成功
            "x-platform-gateway-res-succes": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 服务返回时间
            "x-platform-gateway-res-timestamp": {
                "type": "date"
            },
            // 业务返回码
            "x-platform-res-code": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 服务返回码
            "x-platform-res-message": {
                "type": "text"
            },
            // 服务是否成功
            "x-platform-res-success": {
                "type": "keyword",
                "ignore_above": 256
            },
            // 服务返回时间
            "x-platform-res-timestamp": {
                "type": "date"
            }
        }
    }
}

一个请求报文的示例:

{
  "_index": "data-tinem-test-gateway-request-2023-05-15",
  "_type": "_doc",
  "_id": "92c45a7d-c45f-4883-9c17-f1390081fd55",
  "_version": 1,
  "_score": null,
  "fields": {
    "Origin": [
      ""
    ],
    "x-platform-gateway-req-ip": [
      ""
    ],
    "x-platform-gateway-res-timestamp": [
      "2023-05-15T04:15:46.668Z"
    ],
    "User-Agent": [
      "PostmanRuntime/7.32.2"
    ],
    "x-platform-gateway-res-succes": [
      "true"
    ],
    "x-platform-gateway-req-timestamp": [
      "2023-05-15T04:15:46.306Z"
    ],
    "x-platform-gateway-req-version": [
      "0.0.1-SNAPSHOT"
    ],
    "x-platform-gateway-req-sign": [
      "df6bfe8deeaacf550a87fc376c016715cb1cc3a3c4af5a5f4872c217da95e3d5947d01ed52661e84fd76dbff8deeeee9ea768f6431c85e96284953a0b9ec50a6"
    ],
    "x-platform-gateway-request_id": [
      "92c45a7d-c45f-4883-9c17-f1390081fd55"
    ],
    "x-platform-res-success": [
      "true"
    ],
    "x-platform-gateway-req-sign_type": [
      "SHA512"
    ],
    "x-platform-gateway-req-method": [
      "/event.api"
    ],
    "x-platform-gateway-req-crypto_key": [
      ""
    ],
    "x-platform-gateway-res-message": [
      "success"
    ],
    "x-platform-gateway-req-lang": [
      "en-us"
    ],
    "x-platform-gateway-req-service": [
      "platform-cloud-service-link"
    ],
    "x-platform-res-code": [
      "SUCCESS"
    ],
    "x-platform-gateway-req-charset": [
      "UTF-8"
    ],
    "x-platform-gateway-context-jti": [
      "7ced7a84-9595-4733-a5b0-e4aef3d3a40c"
    ],
    "x-platform-gateway-request-env": [
      "test"
    ],
    "x-platform-gateway-req-crypto_iv": [
      ""
    ],
    "Referer": [
      ""
    ],
    "x-platform-gateway-req-ip-country": [
      ""
    ],
    "Host": [
      "gateway-platform.fuletec.club"
    ],
    "x-platform-gateway-req-jwt": [
      "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOlsiYWxsIiwiYSIsImIiXSwiZ3JhbnRfdHlwZSI6ImNsaWVudF9jcmVkZW50aWFscyIsInNjb3BlIjpbImFsbCJdLCJleHAiOjE2ODQyMDczODIsImp0aSI6IjdjZWQ3YTg0LTk1OTUtNDczMy1hNWIwLWU0YWVmM2QzYTQwYyIsImNsaWVudF9pZCI6ImFkbWluLW5vLXNlY3VyaXR5In0.hIyxJnLRuOJNPalEuFDPa_kbcgyjTVrn_AZG07fuTRS79r2yz2Quk0lsPqUiIKBFo54jdMyLM3n5fwXravM8jQPDrYkvLk6MY6qXQ9vLi1LxVn9O5L0E8Q5gBdCFVjziY06F8KlBD5VaGs4elzt3ekWf1nwZGEqZBM8E49m76g7gFIh5OlrHOF1FPfFzDqRTb1rgHW9DmYJeXFnCEkQfUkt7LyVGT09CU6p4TlBIlPedvIArRdRdtgzZWst6bgyO2lqKFkz8F8XBC1g3rEajStTIjacJqdo8EgYDrL6iSFQMXKGhIXdAwi-65eeavyi_Y3udMEZHgHgYw9spKywEpQ"
    ],
    "x-platform-res-message": [
      "success"
    ],
    "x-platform-gateway-req-api_request_id": [
      "cb516c80-398c-487f-9e48-81f4b191ccb3"
    ],
    "x-platform-gateway-req-crypto_type": [
      ""
    ],
    "x-platform-gateway-req-body": [
      "{\"id\":\"6028648677754028032\",\"ip\":\"222.90.214.224\",\"ua\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36\"}"
    ],
    "x-platform-gateway-accept-timestamp": [
      "2023-05-15T04:15:46.482Z"
    ],
    "x-platform-gateway-context-client_id": [
      "admin-no-security"
    ],
    "x-platform-gateway-req-device": [
      ""
    ],
    "x-platform-gateway-req-debug": [
      ""
    ],
    "x-platform-gateway-res-code": [
      "SUCCESS"
    ]
  },
  "sort": [
    1684124146482
  ]
}

通过kibana监控微服务成功率

  1. 进入创建图表界面

elasticsearch闆嗙兢鏁版嵁瀛樺偍杩囩▼,elasticsearch缃戝叧鑺傜偣

  1. 选择Lens线性图表

elasticsearch闆嗙兢鏁版嵁瀛樺偍杩囩▼,elasticsearch缃戝叧鑺傜偣

  1. 建立图表

elasticsearch闆嗙兢鏁版嵁瀛樺偍杩囩▼,elasticsearch缃戝叧鑺傜偣

这个图表中我们监控各个服务返回错误的数量变化。

通过 x-platform-res-success: false限制只统计错误接口

使用网关收到请求的时间作为X轴

使用统计数量作为Y轴

在Y轴使用x-platform-gateway-req-service做细分

这样就能显示我们每个服务返回错误的数量。

elasticsearch闆嗙兢鏁版嵁瀛樺偍杩囩▼,elasticsearch缃戝叧鑺傜偣

也可以通过Kibana根据业务需要做其他维度的监控或者统计。

如果我们的图标特别复杂还可以通过第三方的图标工具直接从ES中取数据来展示