Initial LisgloSIPS V2 implementation
This commit is contained in:
hectorzhao
@@ -0,0 +1,143 @@
|
||||
# Server A Redis 热路径、HEP 与指标 Runbook
|
||||
|
||||
适用范围：S20 在 Server A 上部署的 OpenSIPS Redis 热路径、HEP 发送、自定义指标和本地 Redis AUTH 代理。
|
||||
|
||||
## 部署结果
|
||||
|
||||
- OpenSIPS 继续监听 `100.90.90.90:15060/udp`，HEP 本地发送 socket 为 `100.90.90.90:9061/udp`。
|
||||
- 本地 Redis AUTH 代理监听 `127.0.0.1:6380/tcp`，上游连接 Server B Redis `100.90.90.91:6379`，凭据通过 systemd encrypted credential 加载，不写入明文配置。
|
||||
- OpenSIPS Redis URL 由预处理器注入为本地代理地址。
|
||||
- Lua 热路径脚本读取版本化配置：
|
||||
- `cfg:active_version`
|
||||
- `cfg:v:{version}:auth:ip:{source_ip}`
|
||||
- `cfg:v:{version}:customer_gateway:{id}`
|
||||
- `cfg:v:{version}:customer:{id}`
|
||||
- `cfg:v:{version}:customer_gateway:{id}:policies`
|
||||
- INVITE 会按 Lua 返回结果降级：
|
||||
- Redis 不可用：`503 Redis Unavailable`
|
||||
- 配置缺失：`503 Config Missing`
|
||||
- 策略缺失：`503 No Route Policy`
|
||||
- 策略允许但 S21/S28 路由未完成：`503 Routing Not Ready`
|
||||
- CDR 冒烟事件写入 `stream:cdr_payload`，S20 验证事件已清理。
|
||||
- Prometheus 指标：
|
||||
- OpenSIPS MI/Prometheus：`http://127.0.0.1:8888/metrics`
|
||||
- Node Exporter textfile：`/var/lib/prometheus/node-exporter/lisglosips_s20.prom`
|
||||
- Server B 抓取 A：`http://100.90.90.90:9100/metrics`
|
||||
|
||||
## 关键文件
|
||||
|
||||
- `/etc/opensips/opensips.cfg`
|
||||
- `/etc/opensips/lisglosips_hotpath.lua`
|
||||
- `/usr/local/sbin/lisglosips-redis-auth-proxy`
|
||||
- `/usr/local/sbin/lisglosips-redis-load-hotpath`
|
||||
- `/usr/local/sbin/lisglosips-opensips-preprocess`
|
||||
- `/usr/local/sbin/lisglosips-s20-metrics`
|
||||
- `/etc/systemd/system/lisglosips-redis-auth-proxy.service`
|
||||
- `/etc/systemd/system/lisglosips-redis-hotpath-load.service`
|
||||
- `/etc/systemd/system/opensips.service.d/20-lisglosips-s20.conf`
|
||||
- `/etc/systemd/system/lisglosips-s20-metrics.service`
|
||||
- `/etc/systemd/system/lisglosips-s20-metrics.timer`
|
||||
- `/etc/default/lisglosips-node-exporter`
|
||||
- `/etc/nftables.d/lisglosips-s20-observability.nft`
|
||||
- `/etc/systemd/system/lisglosips-a-firewall.service`
|
||||
|
||||
仓库副本位于 `infra/server-a/s20/`。
|
||||
|
||||
## 服务检查
|
||||
|
||||
```bash
|
||||
sudo systemctl is-system-running
|
||||
sudo systemctl is-active opensips lisglosips-redis-auth-proxy lisglosips-s20-metrics.timer lisglosips-node-exporter lisglosips-a-firewall
|
||||
sudo systemctl is-enabled opensips lisglosips-redis-auth-proxy lisglosips-redis-hotpath-load lisglosips-s20-metrics.timer lisglosips-node-exporter lisglosips-a-firewall
|
||||
sudo ss -lntup | grep -E ':15060|:9061|:8888|:6380|:9100'
|
||||
```
|
||||
|
||||
`lisglosips-redis-hotpath-load.service` 是 oneshot，执行成功后显示 inactive 属于正常状态；检查 `systemctl show -p Result lisglosips-redis-hotpath-load.service` 应为 `success`。
|
||||
|
||||
## 指标检查
|
||||
|
||||
```bash
|
||||
curl -fsS http://127.0.0.1:8888/metrics | grep 'lisglosips_opensips_s20_'
|
||||
cat /var/lib/prometheus/node-exporter/lisglosips_s20.prom
|
||||
```
|
||||
|
||||
Server B 可抓取：
|
||||
|
||||
```bash
|
||||
curl -fsS http://100.90.90.90:9100/metrics | grep -E 'lisglosips_(voip_rec|recording|redis_proxy|rtpengine)'
|
||||
```
|
||||
|
||||
当前 S20 自定义指标包括：
|
||||
|
||||
- `lisglosips_opensips_s20_invite_total`
|
||||
- `lisglosips_opensips_s20_hotpath_allow_total`
|
||||
- `lisglosips_opensips_s20_hotpath_reject_total`
|
||||
- `lisglosips_opensips_s20_redis_error_total`
|
||||
- `lisglosips_opensips_s20_cdr_xadd_total`
|
||||
- `lisglosips_opensips_s20_cdr_xadd_error_total`
|
||||
- `lisglosips_redis_proxy_up`
|
||||
- `lisglosips_voip_rec_tmpfs_bytes`
|
||||
- `lisglosips_recording_files`
|
||||
- `lisglosips_recording_oldest_ready_age_seconds`
|
||||
- `lisglosips_rtpengine_sessions`
|
||||
|
||||
## HEP 检查
|
||||
|
||||
A 侧确认无发送错误：
|
||||
|
||||
```bash
|
||||
sudo journalctl -u opensips --since '10 minutes ago' --no-pager | grep -E 'Cannot send HEP|Failed to duplicate|S20'
|
||||
```
|
||||
|
||||
B 侧确认 heplify 收包统计：
|
||||
|
||||
```bash
|
||||
journalctl -u heplify-server --since '10 minutes ago' --no-pager | tail -n 80
|
||||
```
|
||||
|
||||
S20 验证中曾因 HEP socket 绑定 `127.0.0.1` 触发发送错误，OpenSIPS 仍按预期返回 SIP 响应；配置修正为 `100.90.90.90:9061` 后，A 侧抓包看到 HEP 发往 B `100.90.90.91:9060`，B heplify 统计显示 HEP 收包且 Error 为 0。
|
||||
|
||||
## 降级验证记录
|
||||
|
||||
- Redis 配置缺失：T 发 INVITE 后 A 返回 `SIP/2.0 503 Config Missing`，OpenSIPS 记录 `S20 hotpath reject reason=CONFIG_MISSING`，CDR XADD 成功。
|
||||
- Redis 代理不可用：短暂停止 `lisglosips-redis-auth-proxy` 后，T 发 INVITE 返回 `SIP/2.0 503 Redis Unavailable`，OpenSIPS 记录 `S20 Redis hotpath unavailable` 和 `S20 CDR XADD failed`；随后启动代理并重新执行 Lua loader 恢复。
|
||||
- HEP 发送异常：绑定错误期间 OpenSIPS 记录 HEP 发送失败，但 SIP 降级响应和 CDR 热路径继续执行；绑定修正后无新增 HEP 发送错误。
|
||||
|
||||
## 回滚
|
||||
|
||||
S20 初始备份目录：
|
||||
|
||||
```text
|
||||
/var/backups/lisglosips-s20/20260621T061900Z
|
||||
```
|
||||
|
||||
后续验证阶段重启 OpenSIPS 前，也在该目录下的 `post-verify/` 保存了当时的 `/etc/opensips/opensips.cfg` 副本。
|
||||
|
||||
回滚步骤：
|
||||
|
||||
1. 停止 S20 新增服务：
|
||||
|
||||
```bash
|
||||
sudo systemctl stop lisglosips-s20-metrics.timer lisglosips-s20-metrics.service
|
||||
sudo systemctl stop lisglosips-redis-auth-proxy.service
|
||||
```
|
||||
|
||||
2. 从备份恢复 `/etc/opensips/opensips.cfg`、OpenSIPS drop-in、Node Exporter 默认文件和 S20 systemd/nftables 文件。
|
||||
3. 执行 OpenSIPS 语法检查：
|
||||
|
||||
```bash
|
||||
sudo opensips -C -f /etc/opensips/opensips.cfg
|
||||
```
|
||||
|
||||
4. 重载 systemd 并重启相关服务：
|
||||
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart opensips
|
||||
sudo systemctl restart lisglosips-node-exporter
|
||||
sudo systemctl restart lisglosips-a-firewall
|
||||
```
|
||||
|
||||
5. 验证 `systemctl is-system-running`、SIP `OPTIONS`、Node Exporter 和 nftables 表。
|
||||
|
||||
回滚不删除 Redis Stream 历史数据；如需清理测试事件，只按明确 Call-ID 精确 `XDEL`。
|
||||
Reference in New Issue
Block a user