Prometheus與傳統(tǒng)進(jìn)程、端口與內(nèi)網(wǎng)域名檢查
背 景
公司A面臨著監(jiān)控其IT基礎(chǔ)設(shè)施的需求,包括Windows和Linux平臺(tái)上的端口、進(jìn)程和內(nèi)網(wǎng)域名狀態(tài)。隨著業(yè)務(wù)的增長(zhǎng),維護(hù)系統(tǒng)的穩(wěn)定性和安全性變得尤為重要。傳統(tǒng)的監(jiān)控方法可能實(shí)時(shí)性和靈活性不夠好,因此采用Prometheus監(jiān)控工具,以便更高效地獲取系統(tǒng)狀態(tài)和性能指標(biāo)。
目 標(biāo)
- 實(shí)現(xiàn)跨平臺(tái)監(jiān)控:選擇合適的監(jiān)控工具,能夠在Windows和Linux上均可安裝和運(yùn)行。
- 實(shí)時(shí)監(jiān)控端口和進(jìn)程:對(duì)關(guān)鍵服務(wù)的端口和進(jìn)程進(jìn)行實(shí)時(shí)監(jiān)控,確保服務(wù)可用性,并及時(shí)告警。
- 內(nèi)網(wǎng)域名狀態(tài)檢查:定期檢查內(nèi)網(wǎng)域名的解析和可達(dá)性,確保內(nèi)部服務(wù)的正常運(yùn)行。
- SSL證書監(jiān)控:自動(dòng)檢查SSL證書的有效性和到期時(shí)間,確保證書時(shí)間健康。
- 數(shù)據(jù)可視化與告警:通過(guò)可視化工具(如Grafana)展示監(jiān)控?cái)?shù)據(jù),并配置告警機(jī)制,以便及時(shí)通知監(jiān)控與開發(fā)相關(guān)技術(shù)人員。
業(yè)務(wù)流程
- 工具選擇與部署
選擇Prometheus作為監(jiān)控系統(tǒng),并利用適配器(如node_exporter、process_exporter、自定義port采集器)來(lái)滿足不同的監(jiān)控需求。
在Windows和Linux服務(wù)器上部署Prometheus及其相關(guān)Exporter。
- 配置端口和進(jìn)程監(jiān)控
使用node_exporter來(lái)監(jiān)控系統(tǒng)的端口和進(jìn)程狀態(tài)。
配置Prometheus以抓取node_exporter提供的指標(biāo),確??梢员O(jiān)控特定的端口和進(jìn)程。
- 告警機(jī)制
采用exporter來(lái)實(shí)現(xiàn)進(jìn)程和端口狀態(tài)檢查。配置HTTP探測(cè),確保服務(wù)的可用性。
將Prometheus數(shù)據(jù)源連接到Grafana,創(chuàng)建儀表盤以可視化監(jiān)控?cái)?shù)據(jù)。
- 測(cè)試與優(yōu)化
在實(shí)施后進(jìn)行測(cè)試,確保監(jiān)控系統(tǒng)能夠準(zhǔn)確捕捉到各項(xiàng)指標(biāo)。
根據(jù)反饋進(jìn)行優(yōu)化,調(diào)整監(jiān)控策略和告警規(guī)則,確保系統(tǒng)的高效運(yùn)行。
Prometheus進(jìn)程、端口配置
①Dockerfile編譯配置
cat Dockerfile
FROM python
ENV LANG=C.UTF-8
ENV TZ=Asia/Shanghai
RUN pip install pyyaml --upgrade -i https://pypi.tuna.tsinghua.edu.cn/simple
RUN pip install requests --upgrade -i https://pypi.tuna.tsinghua.edu.cn/simple
RUN pip install prometheus_client -i https://pypi.tuna.tsinghua.edu.cn/simple
RUN pip install Flask -i https://pypi.tuna.tsinghua.edu.cn/simple
RUN pip install pyyaml -i https://pypi.tuna.tsinghua.edu.cn/simple
RUN pip install asyncio -i https://pypi.tuna.tsinghua.edu.cn/simple
COPY host_port_monitor.py /opt/
CMD ["sleep","999"]
#編譯與推送鏡像到倉(cāng)庫(kù)
docker build -t harbor.export.cn/ops/service_status_monitor_export_port:v2 .
docker push harbor.export.cn/ops/service_status_monitor_export_port:v2
②配置特定端口采集器與端口標(biāo)簽規(guī)范化
cat host_port_monitor.py
# -*- coding:utf-8 -*-
import socket
import os
import yaml
import prometheus_client
from prometheus_client import Gauge
from prometheus_client.core import CollectorRegistry
from flask import Response, Flask
import re
import asyncio
app = Flask(__name__)
def get_config_dic():
"""
Load YAML config file and return it as a dictionary.
"""
pro_path = os.path.dirname(os.path.realpath(__file__))
yaml_path = os.path.join(pro_path, "host_port_conf.yaml")
with open(yaml_path, "r", encoding="utf-8") as f:
sdata = yaml.full_load(f)
return sdata
async def explore_udp_port(ip, port):
try:
loop = asyncio.get_running_loop()
transport, protocol = await loop.create_datagram_endpoint(
lambda: UDPProbe(),
remote_addr=(ip, port)
)
transport.close()
return 1
except Exception:
return 0
class UDPProbe:
def connection_made(self, transport):
self.transport = transport
self.transport.sendto(b'test')
def datagram_received(self, data, addr):
self.transport.close()
def error_received(self, exc):
pass
def connection_lost(self, exc):
pass
def explore_tcp_port(ip, port):
"""
Check if the TCP port is open on the given IP.
"""
try:
tel = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tel.connect((ip, int(port)))
socket.setdefaulttimeout(0.5)
return 1
except:
return 0
def is_valid_label_name(label_name):
"""
Check if the label name is valid according to Prometheus conventions.
"""
return re.match(r'^[a-zA-Z_][a-zA-Z0-9_]*$', label_name) is not None
def format_label_name(label_name):
"""
Format invalid label_name to valid ones by replacing invalid characters.
"""
return re.sub(r'[^a-zA-Z0-9_]', '_', label_name)
def check_port():
"""
Check the ports for all configured services and apply different requirements.
"""
sdic = get_config_dic()
result_list = []
for sertype, config in sdic.items():
iplist = config.get("host")
portlist = config.get("port")
requirement = config.get("requirement")
protocol_list = config.get("protocol", ["tcp"])
# Extract dynamic labels and filter valid ones
dynamic_labels = {key: value for key, value in config.items() if key not in ['host', 'port', 'requirement', 'protocol']}
valid_labels = {format_label_name(key): value for key, value in dynamic_labels.items() if is_valid_label_name(key)}
status_all = True
for ip in iplist:
for port in portlist:
for protocol in protocol_list:
if protocol == "tcp":
status = explore_tcp_port(ip, port)
elif protocol == "udp":
status = asyncio.run(explore_udp_port(ip, port))
else:
status = explore_tcp_port(ip, port)
result_dic = {"sertype": sertype, "host": ip, "port": str(port), "status": status}
# Merge valid dynamic labels into the result dictionary
result_dic.update(valid_labels)
result_list.append(result_dic)
if requirement == "all":
status_all = status_all and result_dic["status"]
elif requirement == "any":
if result_dic["status"]:
status_all = True
break
if requirement in ["all", "any"]:
for result in result_list:
if result["sertype"] == sertype:
result["status"] = int(status_all)
return result_list
@app.route("/metrics")
def api_response():
"""
Generate Prometheus metrics based on the checked ports.
"""
checkport = check_port()
REGISTRY = CollectorRegistry(auto_describe=False)
# Define the metric with labels dynamically
base_labels = ["sertype", "host", "port"]
dynamic_labels = set()
# Collect all unique dynamic labels
for datas in checkport:
dynamic_labels.update(datas.keys())
dynamic_labels = dynamic_labels.difference(base_labels) # Exclude base labels
# Create a Gauge with all valid labels
muxStatus = Gauge("server_port_up", "Api response stats is:", base_labels + list(dynamic_labels), registry=REGISTRY)
for datas in checkport:
# Extract base label values
sertype = datas.get("sertype")
host = datas.get("host")
port = datas.get("port")
status = datas.get("status")
# Prepare label values for dynamic labels
label_values = [sertype, host, port] + [datas.get(label, "unknown") for label in dynamic_labels]
muxStatus.labels(*label_values).set(status)
return Response(prometheus_client.generate_latest(REGISTRY), mimetype="text/plain")
if __name__ == "__main__":
app.run(host="0.0.0.0", port=8080)
#通過(guò)yaml配置端口信息、標(biāo)簽、協(xié)議等等,告警更加直觀。
cat host_port_conf.yaml
# Prometheus monitor server port config.
pw:
env: "prod"
applicationowner: "zhangsan"
applicationname: "zhangsan"
vendor: "export"
techowner: "zhangsan"
service: "pass"
host:
- "10.10.10.123"
- "10.10.10.124"
port:
- 3389
requirement: "check" #正常返回通就通,不通就不通
protocol: "tcp"
ack:
env: "prod"
applicationowner: "zhangsan"
applicationname: "zhangsan"
vendor: "export"
techowner: "zhangsan"
service: "good"
host:
- "10.10.10.128"
- "10.10.10.129"
port:
- 1858
requirement: "any" # 只需滿足一個(gè)就算全通
protocol: "tcp"
#本地服務(wù)測(cè)試查看監(jiān)控特定端口數(shù)據(jù)返回狀態(tài)信息
curl -s -k http://ip:8080/metrics
③在Cronjob配置
cat prometheus-service-monitor-export-port-deploy.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: service-monitor-export-port
namespace: monitor
spec:
replicas: 1
selector:
matchLabels:
app: prometheus-service-monitor-export-port
template:
metadata:
labels:
app: prometheus-service-monitor-export-port
spec:
containers:
- name: service-monitor-export-port
image: harbor.export.cn/ops/service_status_monitor_export_port:v2
imagePullPolicy: Always
command: ["sh","-c"]
args: ["python /opt/host_port_monitor.py"]
resources:
requests:
cpu: 500m
memory: 500Mi
limits:
cpu: 4000m
memory: 4000Mi
ports:
- containerPort: 8080
volumeMounts:
- name: host-port-conf-volume
mountPath: /opt/host_port_conf.yaml
subPath: host_port_conf.yaml
volumes:
- name: host-port-conf-volume
configMap:
name: host-port-conf
---
apiVersion: v1
kind: Service
metadata:
name: service-monitor-export-port
namespace: monitor
spec:
selector:
app: prometheus-service-monitor-export-port
ports:
- name: http
port: 80
targetPort: 8080
type: ClusterIP
cat prometheus-service-monitor-export-port-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: host-port-conf
namespace: monitor
data:
host_port_conf.yaml: |
# Prometheus monitor server port config.
pw:
env: "prod"
applicationowner: "zhangsan"
applicationname: "zhangsan"
vendor: "export"
techowner: "zhangsan"
service: "pass"
host:
- "10.10.10.123"
- "10.10.10.124"
port:
- 3389
requirement: "check" #正常返回通就通,不通就不通
protocol: "tcp"
crm:
env: "prod"
applicationowner: "wangwu"
applicationname: "wangwu"
vendor: "export"
techowner: "wangwu"
service: "good"
host:
- "10.10.10.128"
- "10.10.10.129"
port:
- 1858
requirement: "any" # 只需滿足一個(gè)就算全通
protocol: "tcp"
#推送到ack運(yùn)行該服務(wù)
kubectl apply -f prometheus-service-monitor-export-port-deploy.yaml
kubectl apply -f prometheus-service-monitor-export-port-configmap.yaml
#配置自定義服務(wù)發(fā)現(xiàn)
- job_name: service-status-monitor-export-port
scrape_interval: 30s
scrape_timeout: 30s
scheme: http
metrics_path: /metrics
static_configs:
- targets: ['service-monitor-export-port.monitor.svc:80']
#端口PromQL語(yǔ)句
sum by( host, port, sertype, env, applicationname,applicationowner,techowner,vendor,service,status ) (server_port_up{instance="service-monitor-export-port.monitor.svc:80"}) !=1
④進(jìn)程配置
圖片
Windows進(jìn)程狀態(tài)查詢語(yǔ)句如下:
windows_service_start_mode{instanceIp=~"10.10.10.76|10.10.10.252",start_mode="auto",name=~"mysqld|redis"}
⑤防火墻配置端口策略
如果Windows出現(xiàn)主機(jī)http://10.10.10.22:9400/metrics 為down的,查看Windows的防火墻是否允許9400端口配置,打開cmd命令輸入WF.msc進(jìn)入Windows Defender 防火墻。
圖片
Linux平臺(tái)防火墻配置
#列出系統(tǒng)中的 iptables 規(guī)則,同時(shí)顯示規(guī)則的行號(hào)
iptables -L --line -n
#INPUT和OUTPUT規(guī)則允許9400
iptables -I INPUT 1 -p tcp --dport 9400 -m comment --comment "Allow arms prometheus - 9400 TCP" -j ACCEPT
iptables -I OUTPUT 2 -p tcp --dport 9400 -m comment --comment "Allow arms prometheus - 9400 TCP" -j ACCEPT
#防火墻保存配置
service iptables save
傳統(tǒng)方式進(jìn)程、端口配置
①Linux平臺(tái)配置
可以使用阿里云云助手或者是ansible推送腳本執(zhí)行任務(wù);
#創(chuàng)建腳本執(zhí)行目錄和日志存放目錄
mkdir -p /opt/prot-monitor
mkdir -p /var/log/prot-monitor
監(jiān)控特定進(jìn)程與端口舉例進(jìn)程sshd,端口22;
cat port-monitor.sh
#!/bin/bash
# 設(shè)置要監(jiān)控的服務(wù)
declare -A MONITORS
# 格式: MONITORS["服務(wù)名稱"]="進(jìn)程1,進(jìn)程2:端口1,端口2"
MONITORS["ssh"]="sshd:" # 監(jiān)控進(jìn)程
MONITORS["cyberark"]=":22" # 監(jiān)控端口
# 日志文件路徑
LOG_DIR="/var/log/prot-monitor"
DATE=$(date +"%Y-%m-%d")
PORT_LOG="${LOG_DIR}/port-${DATE}.log"
PROCESS_LOG="${LOG_DIR}/process-${DATE}.log"
# 當(dāng)前時(shí)間
TIMESTAMP=$(date +"%Y-%m-%d %H:%M:%S")
# 獲取主機(jī)名和IP地址
HOSTNAME=$(hostname)
IP_ADDRESS=$(hostname -I | awk '{print $1}') # 取第一個(gè)IP地址
# 清理過(guò)期日志(超過(guò)30天的日志)
find "$LOG_DIR" -name "*.log" -type f -mtime +30 -exec rm -f {} \;
# 檢查端口狀態(tài)函數(shù)
check_port_status() {
local port=$1
local port_status="UP"
# 檢查端口狀態(tài)
local port_list_output=$(netstat -tunlp | grep -v '@pts\|cpus\|master' | awk '{sub(/.*:/,"",$4);sub(/[0-9]*\//,"",$7);print $4}' | sort -n | uniq | egrep -w "$port")
if [[ -z "$port_list_output" ]]; then
port_status="DOWN"
fi
echo $port_status
}
# 檢查進(jìn)程狀態(tài)函數(shù)
check_process_status() {
local process=$1
local process_status="UP"
if ! ps -aux | grep -v grep | grep -q "$process"; then
process_status="DOWN"
fi
echo $process_status
}
# 檢查端口或進(jìn)程狀態(tài)
for SERVICE in "${!MONITORS[@]}"; do
IFS=':' read -r PROCESSES PORTS <<< "${MONITORS[$SERVICE]}"
# 檢查進(jìn)程狀態(tài)
if [[ -n $PROCESSES ]]; then
IFS=',' read -r -a PROCESS_ARRAY <<< "$PROCESSES"
for PROCESS in "${PROCESS_ARRAY[@]}"; do
PROCESS_STATUS=$(check_process_status "$PROCESS")
echo "{\"timestamp\": \"$TIMESTAMP\", \"hostname\": \"$HOSTNAME\", \"ip_address\": \"$IP_ADDRESS\", \"service\": \"$SERVICE\", \"process\": \"$PROCESS\", \"process_status\": \"$PROCESS_STATUS\"}" >> "$PROCESS_LOG"
done
fi
# 檢查端口狀態(tài)
if [[ -n $PORTS ]]; then
IFS=',' read -r -a PORT_ARRAY <<< "$PORTS"
for PORT in "${PORT_ARRAY[@]}"; do
PORT_STATUS=$(check_port_status "$PORT")
echo "{\"timestamp\": \"$TIMESTAMP\", \"hostname\": \"$HOSTNAME\", \"ip_address\": \"$IP_ADDRESS\", \"service\": \"$SERVICE\", \"port_status\": \"$PORT_STATUS\", \"port\": \"$PORT\"}" >> "$PORT_LOG"
done
fi
done
Crontab定時(shí)執(zhí)行任務(wù)計(jì)劃
#每分鐘執(zhí)行腳本port-monitor.sh
crontab -l
*/1 * * * * /bin/bash /opt/prot-monitor/port-monitor.sh
#執(zhí)行查看日志輸出
bash /opt/prot-monitor/port-monitor.sh
#端口日志json輸出
cat /var/log/prot-monitor/port-2024-11-18.log
{"timestamp": "2024-11-18 13:51:01", "hostname": "iZufXXXXXXXXXXXXXXX4Z", "ip_address": "10.10.10.80", "service": "sshd", "port_status": "UP", "port": "22"}
#進(jìn)程日志json輸出
cat /var/log/prot-monitor/process-2024-11-18.log
{"timestamp": "2024-11-18 13:50:01", "hostname": "iZufxxxxxxxxxxxxxx4Z", "ip_address": "10.10.10.80", "service": "sshd", "process": "sshd", "process_status": "UP"}
#查看下進(jìn)程和端口日志服務(wù)返回UP狀態(tài)信息
#啟動(dòng)crond和查看crond狀態(tài)
systemctl start crond
systemctl status crond
#查看crond服務(wù)是否開啟啟動(dòng)
systemctl list-unit-files -t service | grep cron
阿里云SLS對(duì)接服務(wù)器日志路徑采集日志;
#服務(wù)器安裝logtail并運(yùn)行服務(wù)
wget http://logtail-release-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/linux64/logtail.sh -O logtail.sh; chmod +x logtail.sh
#運(yùn)行l(wèi)ogtail服務(wù)
./logtail.sh install auto
sudo /etc/init.d/ilogtaild start
sudo /etc/init.d/ilogtaild status
#查看服務(wù)開機(jī)啟動(dòng)
systemctl list-unit-files -t service | grep ilo
②Windows平臺(tái)配置
監(jiān)控特定進(jìn)程與端口舉例進(jìn)程PM.exe,CA.exe,rdpclip.exe,端口3389
#創(chuàng)建目錄
md C:\monitor-logs
cat C:\monitor-logs\port-monitor.ps1
# 設(shè)置要監(jiān)控的服務(wù)
$MONITORS = @{
"cyberark" = "PM.exe,CA.exe,rdpclip.exe:" # 監(jiān)控進(jìn)程
"rdp" = ":3389" # 監(jiān)控端口
}
# 日志文件路徑
$LOG_DIR = "C:\monitor-logs"
if (!(Test-Path $LOG_DIR)) {
New-Item -Path $LOG_DIR -ItemType Directory | Out-Null
}
$DATE = Get-Date -Format "yyyy-MM-dd"
$PORT_LOG = Join-Path $LOG_DIR "port-$DATE.log"
$PROCESS_LOG = Join-Path $LOG_DIR "process-$DATE.log"
# 當(dāng)前時(shí)間
$TIMESTAMP = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
# 獲取主機(jī)名和IP地址
$HOSTNAME = $env:COMPUTERNAME
$IP_ADDRESS = (Get-NetIPAddress | Where-Object { $_.AddressFamily -eq 'IPv4' -and $_.InterfaceAlias -ne 'Loopback Pseudo-Interface 1' }).IPAddress
# 清理過(guò)期日志(超過(guò)30天的日志)
Get-ChildItem -Path $LOG_DIR -Filter "*.log" | Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-30) } | Remove-Item
# 檢查端口狀態(tài)函數(shù)
function Check-PortStatus {
param (
[int]$port
)
$port_status = "UP"
if (-not (Get-NetTCPConnection -LocalPort $port -ErrorAction SilentlyContinue)) {
$port_status = "DOWN"
}
return $port_status
}
# 使用 tasklist 和 findstr 查詢進(jìn)程狀態(tài)
function Check-ProcessStatus {
param (
[string]$processNameToCheck
)
Write-Host "Checking process: $processNameToCheck"
$process_status = "UP"
# 運(yùn)行 tasklist 并使用 findstr 來(lái)查找進(jìn)程
$tasklist_command = "tasklist /FO CSV /NH | findstr /I `"$processNameToCheck`""
$process_found = Invoke-Expression $tasklist_command
if (-not $process_found) {
Write-Host "Process $processNameToCheck not found."
$process_status = "DOWN"
} else {
Write-Host "Process found: $processNameToCheck"
}
return $process_status
}
# 檢查端口或進(jìn)程狀態(tài)
foreach ($SERVICE in $MONITORS.Keys) {
$entry = $MONITORS[$SERVICE]
$parts = $entry -split ':'
$processes = $parts[0]
$ports = $parts[1]
# 檢查進(jìn)程狀態(tài)
if ($processes) {
$process_array = $processes -split ',' | Where-Object { $_ -ne "" }
foreach ($process in $process_array) {
$process_status = Check-ProcessStatus -processNameToCheck $process
$log_entry = @{
timestamp = $TIMESTAMP
hostname = $HOSTNAME
ip_address = $IP_ADDRESS
service = $SERVICE
process = $process
process_status = $process_status
}
($log_entry | ConvertTo-Json -Compress) | Out-File -Append -FilePath $PROCESS_LOG
}
}
# 檢查端口狀態(tài)
if ($ports) {
$port_array = $ports -split ',' | Where-Object { $_ -ne "" }
foreach ($port in $port_array) {
$port_status = Check-PortStatus -port $port
$log_entry = @{
timestamp = $TIMESTAMP
hostname = $HOSTNAME
ip_address = $IP_ADDRESS
service = $SERVICE
port_status = $port_status
port = $port
}
($log_entry | ConvertTo-Json -Compress) | Out-File -Append -FilePath $PORT_LOG
}
}
}
#查詢特定進(jìn)程信息
tasklist /FO CSV /NH | findstr /I "PM.exe"
tasklist /FO CSV /NH | findstr /I "CA.exe"
③計(jì)劃任務(wù)
cmd打開命令taskschd.msc打開定時(shí)任務(wù);
進(jìn)程和端口監(jiān)控告警每1分鐘運(yùn)行;
圖片
圖片
圖片
圖片
圖片
powershell -WindowStyle Hidden -File "C:\monitor-logs\port-monitor.ps1"
圖片
④SLS進(jìn)程與端口告警配置
#端口查詢語(yǔ)句
*| select timestamp,ip_address,hostname,service,port_status,port from log where port_status is not null and port is not null ORDER BY timestamp DESC limit 1000
#進(jìn)程查詢語(yǔ)句
*| select timestamp,ip_address,hostname,service,process_status from log where process_status is not null ORDER BY timestamp DESC limit 1000
⑤Grafana大屏展示
圖片
內(nèi)網(wǎng)域名狀態(tài)檢查
①Dockerfile配置
FROM python
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
# 指定工作目錄不存會(huì)自己創(chuàng)建
RUN pip install pyyaml --upgrade -i https://pypi.tuna.tsinghua.edu.cn/simple
RUN pip install requests --upgrade -i https://pypi.tuna.tsinghua.edu.cn/simple
COPY domain_return_code.py /opt/
ENV TZ=Asia/Shanghai
# 配置環(huán)境變量信息
CMD ["sleep","999"]
②檢查腳本調(diào)式
#!/usr/bin/env pyhon
# 獲取站點(diǎn)返回狀態(tài)碼
import requests
import yaml
import json
import ssl
import socket
import datetime
def get_cert_expiration_date(host, port):
try:
host = host.split("http://")[1].split("/")[0]
context = ssl.create_default_context()
with socket.create_connection((host, port)) as sock:
with context.wrap_socket(sock, server_hostname=host) as sslsock:
cert = sslsock.getpeercert()
expiration_date = ssl.cert_time_to_seconds(cert['notAfter'])
return expiration_date
except Exception as e:
#print("獲取證書過(guò)期時(shí)間失敗:", e)
#證書不正確返回一個(gè)2008-08-08時(shí)間
return 1218168000
def get_status_code(url,timeout):
try:
requests.packages.urllib3.disable_warnings(requests.packages.urllib3.exceptions.InsecureRequestWarning)
response = requests.get(url,timeout=timeout,verify=False)
response_time = response.elapsed.total_seconds()
#獲取ssl到期時(shí)間
cert_expiration_date = get_cert_expiration_date(url, 443)
#時(shí)間戳轉(zhuǎn)換成年月日
dateArray = datetime.datetime.fromtimestamp(cert_expiration_date, datetime.timezone.utc)
cert_expiration_date_format = dateArray.strftime("%Y-%m-%d")
return response.status_code, response_time, cert_expiration_date_format
except requests.exceptions.RequestException as e:
# print("請(qǐng)求發(fā)生錯(cuò)誤:", e)
return None
def get_domain_returncode(url,timeout):
status_code = get_status_code(url,timeout=timeout)
if status_code is not None:
# print("狀態(tài)碼:", status_code)
return status_code[0], status_code[1], status_code[2]
else:
# print(url + ": " + "504")
return 504,100,'2008-08-08'
def read_yaml(file_path):
with open(file_path, 'r') as file:
try:
yaml_data = yaml.safe_load(file)
return yaml_data
except yaml.YAMLError as e:
print("讀取YAML文件時(shí)發(fā)生錯(cuò)誤:", e)
return None
if __name__ == "__main__":
file_path = "/opt/domain_returncode_conf.yaml"
domain_status_dict = {}
reade_config = read_yaml(file_path)
config = reade_config["domain"]
for i in config:
code = get_domain_returncode(i["Name"],timeout=i["timeout"])
#print('code--------')
#print(code)
# print(i)
for k,v in i.items():
domain_status_dict[k] = v
domain_status_dict["return_code"] = code[0]
domain_status_dict["response_code"] = code[1]
domain_status_dict["EndDate"] = code[2]
json_str = json.dumps(domain_status_dict)
domain_status_dict = {}
print(json_str)
cat /opt/domain_returncode_conf.yaml
domain:
- Name: https://admin.export.cn
timeout: 15
Network: intranet
- Name: https://api.export.cn
timeout: 15
Network: intranet
③Cronjob配置服務(wù)
cat aliyun_domain_code.yaml
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: aliyun-domaincode-monitor-server
namespace: monitor
spec:
schedule: "*/1 * * * *"
concurrencyPolicy: Forbid
jobTemplate:
spec:
parallelism: 1
completions: 1
backoffLimit: 3
activeDeadlineSeconds: 60
ttlSecondsAfterFinished: 600
template:
spec:
volumes:
- name: domain-returncode-config
configMap:
name: domain-returncode
containers:
- name: aliyun-domaincode-monitor
image: harbor.export.cn/ops/domain-return-code:v1
#imagePullSecrets:
imagePullPolicy: Always
command:
- /bin/sh
- -c
- python3 /opt/domain_return_code.py
volumeMounts:
- name: domain-returncode-config
mountPath: /opt/domain_returncode_conf.yaml
subPath: domain_returncode_conf.yaml
restartPolicy: OnFailure
startingDeadlineSeconds: 300
---
apiVersion: v1
kind: ConfigMap
metadata:
name: domain-returncode
namespace: monitoring
data:
domain_returncode_conf.yaml: |
domain:
- Name: https://admin.export.cn
timeout: 15
Network: internet+intranet
environment: prod
vendor: zhangsan
application_name: admin
application_owner: zhangsan
tech_owner: zhangsan
Assignment_group: admin Operations
- Name: https://api.test.cn
timeout: 15
Network: internet
environment: prod
vendor: wangwu
application_name: api
application_owner: test
tech_owner: test
Assignment_group: api Support
④日志輸出返回信息與SLS SQL查詢;
#輸出的日志信息
{"Name": "https://admin.export.cn", "return_code": 200, "response_code": 5.087571, "EndDate": "2025-01-21", "timeout": 15, "Network": "internet+intranet", "environment": "prod", "vendor": "zhangsan", "application_name": "admin", "application_owner": "zhangsan", "tech_owner": "zhangsan", "Assignment_group": "admin Operations"}
SLS查詢語(yǔ)句配置告警;
#查詢證書域名證書時(shí)間
(EndDate: * and _namespace_ : monitoring and _container_name_: aliyun-domaincode-monitor)| select DISTINCT Name,return_code,EndDate,date_diff('day', date_parse(split(_time_, 'T')[1], '%Y-%m-%d'), date_parse(EndDate, '%Y-%m-%d')) as days having days > 0 AND days <= 60 ORDER BY days ASC LIMIT 1000
#查詢5XX的狀態(tài)碼域名返回信息
(((return_code : 5?? ))) not name:"https://view.export.cn" | SELECT DISTINCT Name,return_code,content from log LIMIT 10000
總結(jié)來(lái)說(shuō),公司A通過(guò)采用Prometheus作為監(jiān)控工具,成功地實(shí)現(xiàn)了對(duì)Windows和Linux平臺(tái)上端口、進(jìn)程和內(nèi)網(wǎng)域名狀態(tài)的監(jiān)控。通過(guò)精心設(shè)計(jì)的流程,包括工具選擇與部署、配置端口和進(jìn)程監(jiān)控、告警機(jī)制的建立、以及數(shù)據(jù)可視化和優(yōu)化,公司能夠確保其IT基礎(chǔ)設(shè)施的穩(wěn)定性和安全性。此外,通過(guò)使用Grafana進(jìn)行數(shù)據(jù)可視化和配置告警機(jī)制,公司能夠及時(shí)通知監(jiān)控與開發(fā)相關(guān)技術(shù)人員,從而提高了系統(tǒng)的可用性和響應(yīng)速度。通過(guò)這些措施,公司A能夠有效地管理和維護(hù)其業(yè)務(wù)關(guān)鍵服務(wù),支持業(yè)務(wù)的持續(xù)增長(zhǎng)和擴(kuò)展。