自從上了SkyWalking,睡覺真香?。?!
圖片來自 Pexels
除了應(yīng)用指標監(jiān)控以外,它還能對分布式調(diào)用鏈路進行追蹤。類似功能的組件還有:Zipkin、Pinpoint、CAT 等。
上幾張圖,看看效果,然后再一步一步搭建并使用:
概念與架構(gòu)
SkyWalking 是一個開源監(jiān)控平臺,用于從服務(wù)和云原生基礎(chǔ)設(shè)施收集、分析、聚合和可視化數(shù)據(jù)。
SkyWalking 提供了一種簡單的方法來維護分布式系統(tǒng)的清晰視圖,甚至可以跨云查看。它是一種現(xiàn)代 APM,專門為云原生、基于容器的分布式系統(tǒng)設(shè)計。
SkyWalking 從三個維度對應(yīng)用進行監(jiān)視:
- service(服務(wù))
- service instance(實例)
- endpoint(端點)
服務(wù)和實例就不多說了,端點是服務(wù)中的某個路徑或者說 URI:
SkyWalking allows users to understand the topology relationship between Services and Endpoints, to view the metrics of every Service/Service Instance/Endpoint and to set alarm rules.
SkyWalking 允許用戶了解服務(wù)和端點之間的拓撲關(guān)系,查看每個服務(wù)/服務(wù)實例/端點的度量,并設(shè)置警報規(guī)則。
架構(gòu)如下圖:
SkyWalking 邏輯上分為四個部分:
- Probes(探針)
- Platform backend(平臺后端)
- Storage(存儲)
- UI
這個結(jié)構(gòu)就很清晰了,探針就是 Agent 負責(zé)采集數(shù)據(jù)并上報給服務(wù)端,服務(wù)端對數(shù)據(jù)進行處理和存儲,UI 負責(zé)展示。
下載與安裝
SkyWalking 有兩中版本,ES 版本和非 ES 版。如果我們決定采用 ElasticSearch 作為存儲,那么就下載 ES 版本。
- https://skywalking.apache.org/downloads/
- https://archive.apache.org/dist/skywalking/
如上圖:
- agent 目錄將來要拷貝到各服務(wù)所在機器上用作探針。
- bin 目錄是服務(wù)啟動腳本。
- config 目錄是配置文件。
- oap-libs 目錄是 oap 服務(wù)運行所需的 jar 包。
- webapp 目錄是 web 服務(wù)運行所需的 jar 包。
接下來,要選擇存儲了,支持的存儲有:
- H2
- ElasticSearch 6,7
- MySQL
- TiDB
- InfluxDB
作為監(jiān)控系統(tǒng),首先排除 H2 和 MySQL,這里推薦 InfluxDB,它本身就是時序數(shù)據(jù)庫,非常適合這種場景。但是 InfluxDB 我不是很熟悉,所以這里先用 ElasticSearch7。
- https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md
①安裝 ElasticSearch
鏈接如下:
- https://www.elastic.co/guide/en/elasticsearch/reference/7.10/targz.html
- # 啟動
- ./bin/elasticsearch -d -p pid
- # 停止
- pkill -F pid
ElasticSearch 7.x 需要 Java 11 以上的版本,但是如果你設(shè)置了環(huán)境變量 JAVA_HOME 的話,它會用你自己的 Java 版本。
通常,啟動過程中會報以下三個錯誤:
- [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
- [2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
- [3]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
解決方法:在 /etc/security/limits.conf 文件中追加以下內(nèi)容。
- * soft nofile 65536
- * hard nofile 65536
- * soft nproc 4096
- * hard nproc 4096
可通過以下四個命令查看修改結(jié)果:
- ulimit -Hn
- ulimit -Sn
- ulimit -Hu
- ulimit -Su
修改 /etc/sysctl.conf 文件,追加以下內(nèi)容:
- vm.max_map_count=262144
修改 ES 配置文件 elasticsearch.yml 取消注釋,保留一個節(jié)點:
- cluster.initial_master_nodes: ["node-1"]
為了能夠 ip:port 方式訪問,還需修改網(wǎng)絡(luò)配置:
- network.host: 0.0.0.0
修改完是這樣的:
至此,ElasticSearch 算是啟動成功了。一個節(jié)點還不夠,這里用三個節(jié)點搭建一個集群。
192.168.100.14 config/elasticsearch.yml:
- cluster.name: my-monitor
- node.name: node-1
- network.host: 192.168.100.14
- http.port: 9200
- discovery.seed_hosts: ["192.168.100.14:9300", "192.168.100.15:9300", "192.168.100.19:9300"]
- cluster.initial_master_nodes: ["node-1"]
192.168.100.15 config/elasticsearch.yml:
- cluster.name: my-monitor
- node.name: node-2
- network.host: 192.168.100.15
- http.port: 9200
- discovery.seed_hosts: ["192.168.100.14:9300", "192.168.100.15:9300", "192.168.100.19:9300"]
- cluster.initial_master_nodes: ["node-1"]
192.168.100.19 config/elasticsearch.yml:
- cluster.name: my-monitor
- node.name: node-3
- network.host: 192.168.100.19
- http.port: 9200
- discovery.seed_hosts: ["192.168.100.14:9300", "192.168.100.15:9300", "192.168.100.19:9300"]
- cluster.initial_master_nodes: ["node-1"]
同時,建議修改三個節(jié)點 config/jvm.options:
- -Xms2g
- -Xmx2g
依次啟動三個節(jié)點:
- pkill -F pid
- ./bin/elasticsearch -d -p pid
接下來,修改 skywalking下config/application.yml 中配置 es 地址即可:
- storage:
- selector: ${SW_STORAGE:elasticsearch7}
- elasticsearch7:
- nameSpace: ${SW_NAMESPACE:""}
- clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:192.168.100.14:9200,192.168.100.15:9200,192.168.100.19:9200}
②安裝 Agent
地址如下:
- https://github.com/apache/skywalking/blob/v8.2.0/docs/en/setup/service-agent/java-agent/README.md
將 agent 目錄拷貝至各服務(wù)所在的機器上:
- scp -r ./agent chengjs@192.168.100.12:~/
這里,我將它拷貝至各個服務(wù)目錄下:
plugins 是探針用到各種插件,SkyWalking 插件都是即插即用的,可以把 optional-plugins 中的插件放到 plugins 中。
修改 agent/config/agent.config 配置文件,也可以通過命令行參數(shù)指定。主要是配置服務(wù)名稱和后端服務(wù)地址:
- agent.service_name=${SW_AGENT_NAME:user-center}
- collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:192.168.100.17:11800}
當然,也可以通過環(huán)境變量或系統(tǒng)屬性的方式來設(shè)置,例如:
- export SW_AGENT_COLLECTOR_BACKEND_SERVICES=127.0.0.1:11800
最后,在服務(wù)啟動的時候用命令行參數(shù) -javaagent 來指定探針:
- java -javaagent:/path/to/skywalking-agent/skywalking-agent.jar -jar yourApp.jar
例如:
- java -javaagent:./agent/skywalking-agent.jar -Dspring.profiles.active=dev -Xms512m -Xmx1024m -jar demo-0.0.1-SNAPSHOT.jar
啟動服務(wù)
修改 webapp/webapp.yml 文件,更改端口號及后端服務(wù)地址:
- server:
- port: 9000
- collector:
- path: /graphql
- ribbon:
- ReadTimeout: 10000
- # Point to all backend's restHost:restPort, split by ,
- listOfServers: 127.0.0.1:12800
啟動服務(wù):
- bin/startup.sh
或者分別依次啟動:
- bin/oapService.sh
- bin/webappService.sh
查看 logs 目錄下的日志文件,看是否啟動成功。瀏覽器訪問 :
- http://127.0.0.1:9000
告警
編輯 alarm-settings.yml 設(shè)置告警規(guī)則和通知:
- https://github.com/apache/skywalking/blob/v8.2.0/docs/en/setup/backend/backend-alarm.md
重點說下告警通知:
為了使用釘釘機器人通知,接下來,新建一個項目:
- <?xml version="1.0" encoding="UTF-8"?>
- <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
- <modelVersion>4.0.0</modelVersion>
- <parent>
- <groupId>org.springframework.boot</groupId>
- <artifactId>spring-boot-starter-parent</artifactId>
- <version>2.4.0</version>
- <relativePath/> <!-- lookup parent from repository -->
- </parent>
- <groupId>com.wt.monitor</groupId>
- <artifactId>skywalking-alarm</artifactId>
- <version>1.0.0-SNAPSHOT</version>
- <name>skywalking-alarm</name>
- <properties>
- <java.version>1.8</java.version>
- </properties>
- <dependencies>
- <dependency>
- <groupId>org.springframework.boot</groupId>
- <artifactId>spring-boot-starter-web</artifactId>
- </dependency>
- <dependency>
- <groupId>com.aliyun</groupId>
- <artifactId>alibaba-dingtalk-service-sdk</artifactId>
- <version>1.0.1</version>
- </dependency>
- <dependency>
- <groupId>commons-codec</groupId>
- <artifactId>commons-codec</artifactId>
- <version>1.15</version>
- </dependency>
- <dependency>
- <groupId>com.alibaba</groupId>
- <artifactId>fastjson</artifactId>
- <version>1.2.75</version>
- </dependency>
- <dependency>
- <groupId>org.projectlombok</groupId>
- <artifactId>lombok</artifactId>
- <optional>true</optional>
- </dependency>
- </dependencies>
- <build>
- <plugins>
- <plugin>
- <groupId>org.springframework.boot</groupId>
- <artifactId>spring-boot-maven-plugin</artifactId>
- </plugin>
- </plugins>
- </build>
- </project>
可選依賴(不建議引入):
- <dependency
- <groupId>org.apache.skywalking</groupId>
- <artifactId>server-core</artifactId>
- <version>8.2.0</version>
- </dependency>
定義告警消息實體類:
- package com.wt.monitor.skywalking.alarm.domain;
- import lombok.Data;
- import java.io.Serializable;
- /**
- * @author ChengJianSheng
- * @date 2020/12/1
- */
- @Data
- public class AlarmMessageDTO implements Serializable {
- private int scopeId;
- private String scope;
- /**
- * Target scope entity name
- */
- private String name;
- private String id0;
- private String id1;
- private String ruleName;
- /**
- * Alarm text message
- */
- private String alarmMessage;
- /**
- * Alarm time measured in milliseconds
- */
- private long startTime;
- }
發(fā)送釘釘機器人消息:
- package com.wt.monitor.skywalking.alarm.service;
- import com.dingtalk.api.DefaultDingTalkClient;
- import com.dingtalk.api.DingTalkClient;
- import com.dingtalk.api.request.OapiRobotSendRequest;
- import com.taobao.api.ApiException;
- import lombok.extern.slf4j.Slf4j;
- import org.apache.commons.codec.binary.Base64;
- import org.springframework.beans.factory.annotation.Value;
- import org.springframework.stereotype.Service;
- import javax.crypto.Mac;
- import javax.crypto.spec.SecretKeySpec;
- import java.io.UnsupportedEncodingException;
- import java.net.URLEncoder;
- import java.security.InvalidKeyException;
- import java.security.NoSuchAlgorithmException;
- /**
- * https://ding-doc.dingtalk.com/doc#/serverapi2/qf2nxq
- * @author ChengJianSheng
- * @data 2020/12/1
- */
- @Slf4j
- @Service
- public class DingTalkAlarmService {
- @Value("${dingtalk.webhook}")
- private String webhook;
- @Value("${dingtalk.secret}")
- private String secret;
- public void sendMessage(String content) {
- try {
- Long timestamp = System.currentTimeMillis();
- String stringToSign = timestamp + "\n" + secret;
- Mac mac = Mac.getInstance("HmacSHA256");
- mac.init(new SecretKeySpec(secret.getBytes("UTF-8"), "HmacSHA256"));
- byte[] signData = mac.doFinal(stringToSign.getBytes("UTF-8"));
- String sign = URLEncoder.encode(new String(Base64.encodeBase64(signData)),"UTF-8");
- String serverUrl = webhook + "×tamp=" + timestamp + "&sign=" + sign;
- DingTalkClient client = new DefaultDingTalkClient(serverUrl);
- OapiRobotSendRequest request = new OapiRobotSendRequest();
- request.setMsgtype("text");
- OapiRobotSendRequest.Text text = new OapiRobotSendRequest.Text();
- text.setContent(content);
- request.setText(text);
- client.execute(request);
- } catch (ApiException e) {
- e.printStackTrace();
- log.error(e.getMessage(), e);
- } catch (NoSuchAlgorithmException e) {
- e.printStackTrace();
- log.error(e.getMessage(), e);
- } catch (UnsupportedEncodingException e) {
- e.printStackTrace();
- log.error(e.getMessage(), e);
- } catch (InvalidKeyException e) {
- e.printStackTrace();
- log.error(e.getMessage(), e);
- }
- }
- }
AlarmController.java:
- package com.wt.monitor.skywalking.alarm.controller;
- import com.alibaba.fastjson.JSON;
- import com.wt.monitor.skywalking.alarm.domain.AlarmMessageDTO;
- import com.wt.monitor.skywalking.alarm.service.DingTalkAlarmService;
- import lombok.extern.slf4j.Slf4j;
- import org.springframework.beans.factory.annotation.Autowired;
- import org.springframework.web.bind.annotation.PostMapping;
- import org.springframework.web.bind.annotation.RequestBody;
- import org.springframework.web.bind.annotation.RequestMapping;
- import org.springframework.web.bind.annotation.RestController;
- import java.text.MessageFormat;
- import java.util.List;
- /**
- * @author ChengJianSheng
- * @date 2020/12/1
- */
- @Slf4j
- @RestController
- @RequestMapping("/skywalking")
- public class AlarmController {
- @Autowired
- private DingTalkAlarmService dingTalkAlarmService;
- @PostMapping("/alarm")
- public void alarm(@RequestBody List<AlarmMessageDTO> alarmMessageDTOList) {
- log.info("收到告警信息: {}", JSON.toJSONString(alarmMessageDTOList));
- if (null != alarmMessageDTOList) {
- alarmMessageDTOList.forEach(e->dingTalkAlarmService.sendMessage(MessageFormat.format("-----來自SkyWalking的告警-----\n【名稱】: {0}\n【消息】: {1}\n", e.getName(), e.getAlarmMessage())));
- }
- }
- }
參考文檔:
- https://skywalking.apache.org/
- https://skywalking.apache.org/zh/\ https://github.com/apache/skywalking/tree/v8.2.0/docs
- https://archive.apache.org/dist/
- https://www.elastic.co/guide/en/elasticsearch/reference/master/index.html
- https://www.elastic.co/guide/en/elasticsearch/reference/7.10/modules-discovery-bootstrap-cluster.html
- https://www.elastic.co/guide/en/elasticsearch/reference/7.10/modules-discovery-hosts-providers.html
作者:廢物大師兄
編輯:陶家龍
出處:https://urlify.cn/Zfy2ia