# 节点驱逐 - 完整解决方案 ## 一、环境概览 ### 集群节点信息 | 节点 IP | 角色 | 操作系统 | 状态 | 处理方式 | |---------------|-------------------------------|-----------------------------------------|--------|------------------------| | 10.20.1.130 | controlplane, etcd, worker | openEuler 20.03 (LTS-SP3) | 保留 | master 节点不动 | | 10.20.1.133 | worker | openEuler 20.03 (LTS-SP3) | 保留 | 中间件调度目标 | | 10.20.1.134 | worker | openEuler 20.03 (LTS-SP3) | 保留 | 中间件调度目标 (mysql) | | 10.20.1.141 | worker | BigCloud Enterprise Linux For Euler | 保留 | 中间件调度目标 | | **10.20.1.142** | worker | BigCloud Enterprise Linux For Euler | **清退** | 一个月后清退 | | **10.20.1.144** | worker | BigCloud Enterprise Linux For Euler | **清退** | 一个月后清退 | | **10.20.1.145** | worker | BigCloud Enterprise Linux For Euler | **清退** | 一个月后清退 | ### 需求拆解 1. **清退节点**：10.20.1.142、10.20.1.144、10.20.1.145（一个月后执行） 2. **jxyd 命名空间 Deployment**：生命周期与被清退节点一致 → 即**直接删除** 3. **jxyd 命名空间中间件**：需保留，迁移调度到 10.20.1.133、10.20.1.134、10.20.1.141 4. **资源调整**：超标 Deployment 统一降配 + 同步调整 JVM 参数 --- ## 二、操作流程总览 ``` ┌─────────────────────────────────────────────────────────────────┐ │ Phase 1: 信息采集与备份（立即执行） │ │ - 导出所有 deployment/statefulset 信息 │ │ - 记录当前 Pod 分布 │ │ - 备份所有 YAML │ ├─────────────────────────────────────────────────────────────────┤ │ Phase 2: 资源调整（立即执行） │ │ - 扫描 jxyd 业务 deployment 的资源配置（不扫描中间件） │ │ - 将超标资源统一降配 │ │ - 同步修改 CUST_JAVA_OPTS 环境变量 │ ├─────────────────────────────────────────────────────────────────┤ │ Phase 3: 中间件及业务迁移（立即执行） │ │ - 给保留节点及待清退节点打 label │ │ - 调度中间件至保留节点，调度业务 Deployment 至待清退节点 │ │ - 验证迁移及调度在对应节点上正常运行 │ ├─────────────────────────────────────────────────────────────────┤ │ Phase 4: 节点清退（一个月后执行） │ │ - 删除 jxyd 命名空间中的业务 deployment │ │ - cordon + drain 节点 │ │ - 从集群中移除节点 │ └─────────────────────────────────────────────────────────────────┘ ``` --- ## 三、Phase 1：信息采集与备份 ### 1.1 备份脚本 `phase1_backup.sh` ```bash #!/bin/bash # ============================================================ # Phase 1: 信息采集与备份 # 在 master 节点 (10.20.1.130) 上执行 # ============================================================ BACKUP_DIR="/root/wdd/backup_260617/$(date +%Y%m%d_%H%M%S)" NAMESPACE="jxyd" mkdir -p "${BACKUP_DIR}" echo "==========================================" echo " Phase 1: 信息采集与备份" echo " 备份目录: ${BACKUP_DIR}" echo "==========================================" # 1. 导出节点信息 echo "[1/6] 导出节点信息..." kubectl get nodes -o wide --show-labels > "${BACKUP_DIR}/nodes_info.txt" # 2. 导出 jxyd 命名空间所有资源 echo "[2/6] 导出 jxyd 命名空间所有资源 YAML..." kubectl get all -n ${NAMESPACE} -o yaml > "${BACKUP_DIR}/all_resources.yaml" # 3. 单独导出每个 deployment echo "[3/6] 导出每个 Deployment YAML..." mkdir -p "${BACKUP_DIR}/deployments" for dep in $(kubectl get deployments -n ${NAMESPACE} -o jsonpath='{.items[*].metadata.name}'); do kubectl get deployment "${dep}" -n ${NAMESPACE} -o yaml > "${BACKUP_DIR}/deployments/${dep}.yaml" echo " 已备份 deployment: ${dep}" done # 4. 单独导出每个 statefulset（中间件通常用 statefulset） echo "[4/6] 导出每个 StatefulSet YAML..." mkdir -p "${BACKUP_DIR}/statefulsets" for sts in $(kubectl get statefulsets -n ${NAMESPACE} -o jsonpath='{.items[*].metadata.name}' 2>/dev/null); do kubectl get statefulset "${sts}" -n ${NAMESPACE} -o yaml > "${BACKUP_DIR}/statefulsets/${sts}.yaml" echo " 已备份 statefulset: ${sts}" done # 5. 导出 ConfigMap 和 Secret echo "[5/6] 导出 ConfigMap 和 Secret..." mkdir -p "${BACKUP_DIR}/configmaps" "${BACKUP_DIR}/secrets" for cm in $(kubectl get configmaps -n ${NAMESPACE} -o jsonpath='{.items[*].metadata.name}'); do kubectl get configmap "${cm}" -n ${NAMESPACE} -o yaml > "${BACKUP_DIR}/configmaps/${cm}.yaml" done for sec in $(kubectl get secrets -n ${NAMESPACE} -o jsonpath='{.items[*].metadata.name}'); do kubectl get secret "${sec}" -n ${NAMESPACE} -o yaml > "${BACKUP_DIR}/secrets/${sec}.yaml" done # 6. 导出 Pod 分布信息（关键：记录哪些 Pod 在待清退节点上） echo "[6/6] 导出 Pod 分布信息..." echo "=== 所有 Pod 的节点分布 ===" > "${BACKUP_DIR}/pod_distribution.txt" kubectl get pods -n ${NAMESPACE} -o wide >> "${BACKUP_DIR}/pod_distribution.txt" echo "" echo "--- 待清退节点上的 Pod ---" >> "${BACKUP_DIR}/pod_distribution.txt" for NODE in 10.20.1.142 10.20.1.144 10.20.1.145; do echo "" >> "${BACKUP_DIR}/pod_distribution.txt" echo "=== 节点 ${NODE} 上的 Pod ===" >> "${BACKUP_DIR}/pod_distribution.txt" kubectl get pods -n ${NAMESPACE} -o wide --field-selector spec.nodeName=${NODE} >> "${BACKUP_DIR}/pod_distribution.txt" done # 7. 导出 Service 和 Ingress mkdir -p "${BACKUP_DIR}/services" "${BACKUP_DIR}/ingresses" for svc in $(kubectl get svc -n ${NAMESPACE} -o jsonpath='{.items[*].metadata.name}'); do kubectl get svc "${svc}" -n ${NAMESPACE} -o yaml > "${BACKUP_DIR}/services/${svc}.yaml" done for ing in $(kubectl get ingress -n ${NAMESPACE} -o jsonpath='{.items[*].metadata.name}' 2>/dev/null); do kubectl get ingress "${ing}" -n ${NAMESPACE} -o yaml > "${BACKUP_DIR}/ingresses/${ing}.yaml" done # 8. 生成资源使用汇总 echo "" echo "=== 资源使用汇总 ===" | tee "${BACKUP_DIR}/resource_summary.txt" kubectl top nodes 2>/dev/null | tee -a "${BACKUP_DIR}/resource_summary.txt" echo "" | tee -a "${BACKUP_DIR}/resource_summary.txt" kubectl top pods -n ${NAMESPACE} 2>/dev/null | tee -a "${BACKUP_DIR}/resource_summary.txt" echo "" echo "==========================================" echo " 备份完成！目录: ${BACKUP_DIR}" echo "==========================================" ``` --- ## 四、Phase 2：资源检查与调整 ### 2.1 Python 脚本 `phase2_resource_adjust.py` 此脚本会： - 扫描 jxyd 命名空间下所有 Deployment（中间件不存在资源限制，且只在 StatefulSet，因此阶段二不考虑中间件） - 检测哪些容器的资源超标（超过 limits cpu:2 / memory:2Gi） - 对超标的 Deployment 自动执行 `kubectl patch` 降配 - 同步修改 `CUST_JAVA_OPTS` 环境变量 ```python #!/usr/bin/env python3 # -*- coding: utf-8 -*- """ Phase 2: 资源检查与调整 功能： 1. 扫描 jxyd 命名空间下所有 Deployment 的资源配置 2. 识别超标资源（超过 limits cpu:2 / memory:2Gi） 3. 自动 patch 降配 + 修改 CUST_JAVA_OPTS 4. 生成变更报告 使用方式： python3 phase2_resource_adjust.py --dry-run # 仅预览，不执行变更 python3 phase2_resource_adjust.py --apply # 执行变更 """ import json import subprocess import sys import re import argparse from datetime import datetime # ============================================================ # 配置区 # ============================================================ NAMESPACE = "jxyd" # 目标资源规格 TARGET_RESOURCES = { "limits": {"cpu": "2", "memory": "2Gi"}, "requests": {"cpu": "1", "memory": "500Mi"} } # 目标 JVM 参数 TARGET_JAVA_OPTS = "-Xms500m -Xmx2000m -Dlog4j2.formatMsgNoLookups=true" # ============================================================ # 工具函数 # ============================================================ def run_kubectl(args, capture=True): """执行 kubectl 命令""" cmd = ["kubectl"] + args result = subprocess.run(cmd, capture_output=capture, text=True) if result.returncode != 0: print(f" [错误] 命令失败: {' '.join(cmd)}") print(f" stderr: {result.stderr}") return None return result.stdout def parse_memory(mem_str): """将内存字符串转换为字节数""" if not mem_str: return 0 mem_str = str(mem_str) units = { 'Ki': 1024, 'Mi': 1024**2, 'Gi': 1024**3, 'Ti': 1024**4, 'K': 1000, 'M': 1000**2, 'G': 1000**3, 'T': 1000**4, 'k': 1000, 'm': 0.001 # millibytes (unusual but valid) } for suffix, multiplier in sorted(units.items(), key=lambda x: -len(x[0])): if mem_str.endswith(suffix): try: return float(mem_str[:-len(suffix)]) * multiplier except ValueError: return 0 try: return float(mem_str) # 纯数字，单位为字节 except ValueError: return 0 def parse_cpu(cpu_str): """将 CPU 字符串转换为核心数（float）""" if not cpu_str: return 0.0 cpu_str = str(cpu_str) if cpu_str.endswith('m'): return float(cpu_str[:-1]) / 1000.0 return float(cpu_str) def is_resource_over_limit(resources): """检查资源是否超标""" limits = resources.get("limits", {}) cpu_limit = limits.get("cpu", "0") mem_limit = limits.get("memory", "0") cpu_over = parse_cpu(cpu_limit) > parse_cpu(TARGET_RESOURCES["limits"]["cpu"]) mem_over = parse_memory(mem_limit) > parse_memory(TARGET_RESOURCES["limits"]["memory"]) return cpu_over or mem_over # ============================================================ # 主逻辑 # ============================================================ def get_deployments(): """获取所有 Deployment 信息""" output = run_kubectl([ "get", "deployments", "-n", NAMESPACE, "-o", "json" ]) if not output: return [] data = json.loads(output) return data.get("items", []) def analyze_and_patch(workloads, kind, dry_run=True): """分析并 patch 超标资源，若副本数为 0 则直接删除""" report = { "business": [], "patched": [], "skipped": [], "errors": [], "deleted": [] } for wl in workloads: name = wl["metadata"]["name"] replicas = wl["spec"].get("replicas", 1) if replicas == 0: print(f"\n [删除] {kind}/{name} (副本数为 0)") if dry_run: print(" [DRY-RUN] 跳过删除") report["skipped"].append({"name": name, "reason": "replicas=0"}) else: result = run_kubectl([ "delete", kind.lower(), name, "-n", NAMESPACE ]) if result is not None: print(" [成功] 已删除") report["deleted"].append(name) else: print(" [失败] 删除失败") report["errors"].append({"name": name, "reason": "delete failed"}) continue spec = wl["spec"]["template"]["spec"] containers = spec.get("containers", []) category = "business" for idx, container in enumerate(containers): container_name = container.get("name", f"container-{idx}") resources = container.get("resources", {}) limits = resources.get("limits", {}) requests = resources.get("requests", {}) # 提取当前 JAVA_OPTS current_java_opts = "" env_list = container.get("env", []) for env in env_list: if env.get("name") == "CUST_JAVA_OPTS": current_java_opts = env.get("value", "") break item = { "name": name, "kind": kind, "container": container_name, "container_index": idx, "category": category, "current_limits_cpu": limits.get("cpu", "未设置"), "current_limits_memory": limits.get("memory", "未设置"), "current_requests_cpu": requests.get("cpu", "未设置"), "current_requests_memory": requests.get("memory", "未设置"), "current_java_opts": current_java_opts, "over_limit": is_resource_over_limit(resources), } report[category].append(item) # 只调整超标的 if item["over_limit"]: print(f"\n [超标] {kind}/{name} (容器: {container_name})") print(f" 当前: limits={{cpu:{limits.get('cpu', 'N/A')}, memory:{limits.get('memory', 'N/A')}}}") print(f" 目标: limits={{cpu:{TARGET_RESOURCES['limits']['cpu']}, memory:{TARGET_RESOURCES['limits']['memory']}}}") if dry_run: print(f" [DRY-RUN] 跳过 patch") report["skipped"].append(item) else: # 构造 patch JSON patch = { "spec": { "template": { "spec": { "containers": [] } } } } # 构建容器 patch（需要按索引） container_patch = { "name": container_name, "resources": TARGET_RESOURCES } # 同步修改 CUST_JAVA_OPTS（如果存在该 env） if current_java_opts: # 使用 kubectl set env 更简洁，这里用 strategic merge patch new_env = [] for env in env_list: if env.get("name") == "CUST_JAVA_OPTS": new_env.append({ "name": "CUST_JAVA_OPTS", "value": TARGET_JAVA_OPTS }) else: new_env.append(env) container_patch["env"] = new_env patch["spec"]["template"]["spec"]["containers"] = [container_patch] patch_json = json.dumps(patch) result = run_kubectl([ "patch", kind.lower(), name, "-n", NAMESPACE, "--type", "strategic", "-p", patch_json ]) if result is not None: print(f" [成功] 已 patch {kind}/{name}") report["patched"].append(item) else: print(f" [失败] patch {kind}/{name} 失败") report["errors"].append(item) else: # 未超标但可能也需要同步 JAVA_OPTS if current_java_opts and current_java_opts != TARGET_JAVA_OPTS: print(f"\n [JAVA_OPTS 不一致] {kind}/{name} (容器: {container_name})") print(f" 当前: {current_java_opts}") print(f" 目标: {TARGET_JAVA_OPTS}") if not dry_run: result = run_kubectl([ "set", "env", f"{kind.lower()}/{name}", "-n", NAMESPACE, f"CUST_JAVA_OPTS={TARGET_JAVA_OPTS}", "-c", container_name ]) if result is not None: print(f" [成功] 已更新 JAVA_OPTS") else: print(f" [失败] 更新 JAVA_OPTS 失败") return report def print_summary(report, kind): """打印汇总报告""" print(f"\n{'='*60}") print(f" {kind} 资源扫描报告") print(f"{'='*60}") print(f"\n 🚀 业务 Deployment ({len(report['business'])} 个容器):") for item in report["business"]: over = "⚠️ 超标" if item["over_limit"] else "✅ 正常" print(f" - {item['name']}/{item['container']}: " f"limits(cpu={item['current_limits_cpu']}, mem={item['current_limits_memory']}) " f"requests(cpu={item['current_requests_cpu']}, mem={item['current_requests_memory']}) " f"{over}") if report.get("deleted"): print(f"\n 🗑️ 已删除 (副本数为0): {len(report['deleted'])} 个") if report["patched"]: print(f"\n 🔧 已 Patch: {len(report['patched'])} 个") if report["skipped"]: print(f" ⏭️ 跳过 (DRY-RUN): {len(report['skipped'])} 个") if report["errors"]: print(f" ❌ 失败: {len(report['errors'])} 个") def main(): parser = argparse.ArgumentParser(description="Phase 2: jxyd 命名空间资源检查与调整") group = parser.add_mutually_exclusive_group(required=True) group.add_argument("--dry-run", action="store_true", help="仅预览，不执行变更") group.add_argument("--apply", action="store_true", help="执行变更") args = parser.parse_args() dry_run = args.dry_run print(f"\n{'#'*60}") print(f" Phase 2: 资源检查与调整") print(f" 命名空间: {NAMESPACE}") print(f" 模式: {'DRY-RUN（仅预览）' if dry_run else '⚡ APPLY（执行变更）'}") print(f" 时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}") print(f"{'#'*60}") if not dry_run: confirm = input("\n ⚠️ 确认要执行资源调整吗？(yes/no): ") if confirm.lower() != "yes": print(" 已取消。") return # 扫描 Deployment (业务) print(f"\n{'='*60}") print(" 扫描 Deployment...") print(f"{'='*60}") deployments = get_deployments() dep_report = analyze_and_patch(deployments, "Deployment", dry_run) print_summary(dep_report, "Deployment") # 生成变更日志 log_file = f"/root/wdd/backup_260617/phase2_report_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json" log_data = { "timestamp": datetime.now().isoformat(), "mode": "dry-run" if dry_run else "apply", "deployment_report": { "business_count": len(dep_report["business"]), "patched_count": len(dep_report["patched"]), "error_count": len(dep_report["errors"]), } } try: with open(log_file, 'w') as f: json.dump(log_data, f, indent=2, ensure_ascii=False) print(f"\n 📄 报告已保存到: {log_file}") except Exception as e: print(f"\n [警告] 无法保存报告文件: {e}") print(f"\n{'#'*60}") print(f" Phase 2 完成!") print(f"{'#'*60}") if __name__ == "__main__": main() ``` --- ## 五、Phase 3：中间件迁移 ### 3.1 给保留节点打 Label ```bash #!/bin/bash # ============================================================ # Phase 3.1: 给保留节点打 Label # ============================================================ echo "=== 给保留节点打 label: jxyd-middleware=true ===" # 保留节点 kubectl label node 10.20.1.133 jxyd-middleware=true --overwrite kubectl label node 10.20.1.134 jxyd-middleware=true --overwrite kubectl label node 10.20.1.141 jxyd-middleware=true --overwrite echo "" echo "" echo "=== 给待清退节点打 label: jxyd-business=true ===" # 待清退节点（将业务部署到这些节点） kubectl label node 10.20.1.142 jxyd-business=true --overwrite kubectl label node 10.20.1.144 jxyd-business=true --overwrite kubectl label node 10.20.1.145 jxyd-business=true --overwrite echo "" echo "=== 验证 label ===" kubectl get nodes -l jxyd-middleware=true -o wide kubectl get nodes -l jxyd-business=true -o wide ``` ### 3.2 调度与迁移脚本 `phase3_migrate_workloads.py` ```python #!/usr/bin/env python3 # -*- coding: utf-8 -*- """ Phase 3: 工作负载迁移与调度 功能： 1. 识别 jxyd 命名空间下的中间件 StatefulSet 与 业务 Deployment 2. 为中间件添加 nodeSelector，调度到保留节点（过滤包含 hostPath 的中间件） 3. 为业务 Deployment 添加 nodeSelector，调度到待清退节点 4. 验证调度和迁移结果 使用方式： python3 phase3_migrate_workloads.py --list # 列出迁移计划 python3 phase3_migrate_workloads.py --dry-run # 预览变更 python3 phase3_migrate_workloads.py --apply # 执行迁移 python3 phase3_migrate_workloads.py --verify # 验证迁移结果 """ import json import subprocess import sys import argparse from datetime import datetime NAMESPACE = "jxyd" # 目标节点 label TARGET_NODE_SELECTOR = {"jxyd-middleware": "true"} BUSINESS_NODE_SELECTOR = {"jxyd-business": "true"} # 中间件关键词 MIDDLEWARE_KEYWORDS = [ "mysql", "redis", "rabbitmq", "kafka", "zookeeper", "elasticsearch", "nacos", "minio", "mongo", "postgres", "nginx", "sentinel", "rocketmq", "emqx", "mqtt", "influxdb", "grafana", "prometheus", "xxl-job", "seata" ] # 待清退节点 EVICT_NODES = ["10.20.1.142", "10.20.1.144", "10.20.1.145"] def run_kubectl(args): cmd = ["kubectl"] + args result = subprocess.run(cmd, capture_output=True, text=True) if result.returncode != 0: print(f" [错误] {' '.join(cmd)}: {result.stderr.strip()}") return None return result.stdout def is_middleware(name): name_lower = name.lower() return any(kw in name_lower for kw in MIDDLEWARE_KEYWORDS) def get_workloads(kind): output = run_kubectl(["get", kind, "-n", NAMESPACE, "-o", "json"]) if not output: return [] return json.loads(output).get("items", []) def list_workloads(): """列出工作负载迁移与调度计划""" print(f"\n{'='*60}") print(f" jxyd 命名空间 - 迁移与调度计划") print(f"{'='*60}") # StatefulSet (中间件) sts_workloads = get_workloads("statefulsets") mw_list = [w for w in sts_workloads if is_middleware(w["metadata"]["name"])] print(f"\n 📦 中间件 StatefulSet (调度到保留节点，{len(mw_list)} 个):") for w in mw_list: name = w["metadata"]["name"] replicas = w["spec"].get("replicas", 1) node_selector = w["spec"]["template"]["spec"].get("nodeSelector", {}) volumes = w["spec"]["template"]["spec"].get("volumes", []) has_host_path = any("hostPath" in v for v in volumes) host_path_flag = "[含 hostPath，不可迁移]" if has_host_path else "" print(f" ✅ {name} (replicas={replicas}, nodeSelector={node_selector}) {host_path_flag}") # Deployment (业务) dep_workloads = get_workloads("deployments") print(f"\n 🚀 业务 Deployment (调度到待清退节点，{len(dep_workloads)} 个):") for w in dep_workloads: name = w["metadata"]["name"] replicas = w["spec"].get("replicas", 1) node_selector = w["spec"]["template"]["spec"].get("nodeSelector", {}) if replicas == 0: print(f" - {name} (replicas=0, 忽略调度)") else: print(f" ✅ {name} (replicas={replicas}, nodeSelector={node_selector})") print(f"\n 💡 提示: 若状态不准确可人工审核配置。") def migrate_workloads(dry_run=True): """执行工作负载的迁移与调度""" print(f"\n{'='*60}") print(f" 工作负载调度 - {'DRY-RUN' if dry_run else 'APPLY'}") print(f"{'='*60}") patched = 0 errors = 0 def apply_patch(kind, w, target_ns, is_middleware_check=False): nonlocal patched, errors name = w["metadata"]["name"] replicas = w["spec"].get("replicas", 1) current_ns = w["spec"]["template"]["spec"].get("nodeSelector", {}) if replicas == 0: return # 忽略副本数为 0 的 if is_middleware_check: volumes = w["spec"]["template"]["spec"].get("volumes", []) has_host_path = any("hostPath" in v for v in volumes) if has_host_path: print(f" [跳过] {kind}/{name} 包含 hostPath 挂载，不能进行迁移") return # 检查是否已包含所有的目标 selector if all(current_ns.get(k) == v for k, v in target_ns.items()): print(f" [跳过] {kind}/{name} 已满足 nodeSelector 目标") return print(f" [调度] {kind}/{name}") print(f" 当前 nodeSelector: {current_ns}") print(f" 目标 nodeSelector: {target_ns}") if dry_run: print(f" [DRY-RUN] 跳过") return merged_selector = {**current_ns, **target_ns} patch = {"spec": {"template": {"spec": {"nodeSelector": merged_selector}}}} result = run_kubectl([ "patch", kind, name, "-n", NAMESPACE, "--type", "strategic", "-p", json.dumps(patch) ]) if result is not None: print(f" [成功] 已 patch") patched += 1 else: print(f" [失败]") errors += 1 # 处理中间件 sts_workloads = get_workloads("statefulsets") for w in [w for w in sts_workloads if is_middleware(w["metadata"]["name"])]: apply_patch("statefulset", w, TARGET_NODE_SELECTOR, True) # 处理业务 dep_workloads = get_workloads("deployments") for w in dep_workloads: apply_patch("deployment", w, BUSINESS_NODE_SELECTOR, False) print(f"\n 汇总: 已调度 {patched} 个, 失败 {errors} 个") def verify_migration(): """验证迁移及调度结果""" print(f"\n{'='*60}") print(f" 调度结果验证") print(f"{'='*60}") issues = [] def check_workloads(kind, workloads, target_label_key, target_nodes_list, is_middleware_check=False): for w in workloads: name = w["metadata"]["name"] replicas = w["spec"].get("replicas", 1) if replicas == 0: continue ns = w["spec"]["template"]["spec"].get("nodeSelector", {}) if is_middleware_check: volumes = w["spec"]["template"]["spec"].get("volumes", []) if any("hostPath" in v for v in volumes): continue if ns.get(target_label_key) != "true": issues.append(f" ⚠️ {kind}/{name} 未设置 {target_label_key} nodeSelector") pods_output = run_kubectl(["get", "pods", "-n", NAMESPACE, "-l", f"app={name}", "-o", "json"]) if not pods_output or json.loads(pods_output).get("items", []) == []: pods_output = run_kubectl(["get", "pods", "-n", NAMESPACE, "-o", "json"]) if pods_output: all_pods = [p for p in json.loads(pods_output).get("items", []) if p["metadata"]["name"].startswith(name)] else: all_pods = [] else: all_pods = json.loads(pods_output).get("items", []) for pod in all_pods: pod_name = pod["metadata"]["name"] node = pod["spec"].get("nodeName", "unknown") phase = pod["status"].get("phase", "Unknown") # Check target nodes if target_nodes_list is not None and node not in target_nodes_list: issues.append(f" ⚠️ Pod {pod_name} 调度异常，当前节点 {node}，应在目标节点集合 {target_nodes_list} 中") # Exclude nodes elif target_nodes_list is None and node in EVICT_NODES: issues.append(f" ⚠️ Pod {pod_name} 仍在待清退节点 {node} 上") else: print(f" ✅ {pod_name} -> 节点 {node} (状态: {phase})") # 1. 中间件需在保留节点上（非待清退节点） sts_workloads = get_workloads("statefulsets") mw_list = [w for w in sts_workloads if is_middleware(w["metadata"]["name"])] check_workloads("statefulset", mw_list, "jxyd-middleware", None, True) # 2. 业务需在待清退节点上 dep_workloads = get_workloads("deployments") check_workloads("deployment", dep_workloads, "jxyd-business", EVICT_NODES, False) if issues: print(f"\n ⚠️ 发现 {len(issues)} 个问题:") for issue in issues: print(issue) else: print(f"\n ✅ 所有的工作负载已成功调度到对应的目标节点!") def main(): parser = argparse.ArgumentParser(description="Phase 3: 工作负载迁移与调度") group = parser.add_mutually_exclusive_group(required=True) group.add_argument("--list", action="store_true", help="列出迁移计划") group.add_argument("--dry-run", action="store_true", help="预览变更") group.add_argument("--apply", action="store_true", help="执行迁移") group.add_argument("--verify", action="store_true", help="验证迁移结果") args = parser.parse_args() if args.list: list_workloads() elif args.dry_run: migrate_workloads(dry_run=True) elif args.apply: confirm = input(" ⚠️ 确认要执行迁移和调度吗？(yes/no): ") if confirm.lower() == "yes": migrate_workloads(dry_run=False) else: print(" 已取消。") elif args.verify: verify_migration() if __name__ == "__main__": main() ``` --- ## 六、Phase 4：节点清退（一个月后执行） ### 4.1 清退脚本 `phase4_evict_nodes.sh` ```bash #!/bin/bash # ============================================================ # Phase 4: 节点清退（一个月后执行） # # ⚠️ 此脚本将执行不可逆操作，请确认： # 1. Phase 2 资源调整已完成 # 2. Phase 3 中间件迁移已完成并验证 # 3. 已获得相关审批 # ============================================================ set -e NAMESPACE="jxyd" EVICT_NODES=("10.20.1.142" "10.20.1.144" "10.20.1.145") LOG_DIR="/root/wdd/backup_260617/phase4_$(date +%Y%m%d_%H%M%S)" mkdir -p "${LOG_DIR}" echo "==========================================" echo " Phase 4: 节点清退" echo " 时间: $(date '+%Y-%m-%d %H:%M:%S')" echo " 日志: ${LOG_DIR}" echo "==========================================" # ============================================================ # Step 1: 最终确认 # ============================================================ echo "" echo "⚠️ 即将执行以下操作：" echo " 1. 删除 ${NAMESPACE} 命名空间中的业务 Deployment" echo " 2. Cordon 节点: ${EVICT_NODES[*]}" echo " 3. Drain 节点: ${EVICT_NODES[*]}" echo " 4. 从集群中删除节点: ${EVICT_NODES[*]}" echo "" read -p "确认执行？(输入 YES 继续): " CONFIRM if [ "${CONFIRM}" != "YES" ]; then echo "已取消。" exit 0 fi # ============================================================ # Step 2: 最终备份 # ============================================================ echo "" echo "=== Step 2: 最终备份 ===" kubectl get all -n ${NAMESPACE} -o yaml > "${LOG_DIR}/final_backup_all.yaml" kubectl get pods -n ${NAMESPACE} -o wide > "${LOG_DIR}/final_pod_distribution.txt" echo " 备份完成" # ============================================================ # Step 3: 删除 jxyd 业务 Deployment # （说明：中间件仅存在于 StatefulSet，因此所有的 Deployment 均为业务） # ============================================================ echo "" echo "=== Step 3: 删除 jxyd 业务 Deployment ===" # 获取所有 deployment ALL_DEPS=$(kubectl get deployments -n ${NAMESPACE} -o jsonpath='{.items[*].metadata.name}') for dep in ${ALL_DEPS}; do echo " [删除-业务] ${dep}" kubectl delete deployment "${dep}" -n ${NAMESPACE} --grace-period=30 2>&1 | tee -a "${LOG_DIR}/delete_deployments.log" done # ============================================================ # Step 4: Cordon 节点（标记不可调度） # ============================================================ echo "" echo "=== Step 4: Cordon 节点 ===" for NODE in "${EVICT_NODES[@]}"; do echo " Cordon ${NODE}..." kubectl cordon "${NODE}" 2>&1 | tee -a "${LOG_DIR}/cordon.log" done echo "" kubectl get nodes -o wide | tee "${LOG_DIR}/nodes_after_cordon.txt" # ============================================================ # Step 5: Drain 节点（驱逐所有 Pod） # ============================================================ echo "" echo "=== Step 5: Drain 节点 ===" for NODE in "${EVICT_NODES[@]}"; do echo " Drain ${NODE}..." kubectl drain "${NODE}" \ --ignore-daemonsets \ --delete-emptydir-data \ --force \ --grace-period=60 \ --timeout=300s \ 2>&1 | tee -a "${LOG_DIR}/drain_${NODE}.log" echo " 等待 30 秒让 Pod 完成迁移..." sleep 30 done # ============================================================ # Step 6: 验证 Pod 已全部迁走 # ============================================================ echo "" echo "=== Step 6: 验证 Pod 迁移情况 ===" for NODE in "${EVICT_NODES[@]}"; do REMAINING=$(kubectl get pods --all-namespaces --field-selector spec.nodeName=${NODE} --no-headers 2>/dev/null | grep -v "kube-system" | wc -l) if [ "${REMAINING}" -gt 0 ]; then echo " ⚠️ 节点 ${NODE} 上仍有 ${REMAINING} 个非系统 Pod:" kubectl get pods --all-namespaces --field-selector spec.nodeName=${NODE} -o wide | grep -v "kube-system" else echo " ✅ 节点 ${NODE} 上的用户 Pod 已全部迁走" fi done # ============================================================ # Step 7: 从集群中删除节点 # ============================================================ echo "" echo "=== Step 7: 从集群中删除节点 ===" read -p "确认从集群中删除节点？(输入 YES 继续): " CONFIRM2 if [ "${CONFIRM2}" == "YES" ]; then for NODE in "${EVICT_NODES[@]}"; do echo " 删除节点 ${NODE}..." kubectl delete node "${NODE}" 2>&1 | tee -a "${LOG_DIR}/delete_nodes.log" done else echo " 跳过节点删除。可稍后手动执行: kubectl delete node " fi # ============================================================ # Step 8: 最终验证 # ============================================================ echo "" echo "=== Step 8: 最终验证 ===" echo "--- 集群节点状态 ---" kubectl get nodes -o wide | tee "${LOG_DIR}/nodes_final.txt" echo "" echo "--- jxyd 命名空间 Pod 状态 ---" kubectl get pods -n ${NAMESPACE} -o wide | tee "${LOG_DIR}/pods_final.txt" echo "" echo "==========================================" echo " Phase 4 完成！" echo " 日志目录: ${LOG_DIR}" echo "==========================================" ``` --- ## 七、快速命令参考 ### 如果需要手动单独操作 ```bash # ============================================================ # 常用单条命令（按需执行） # ============================================================ # 1. 查看 jxyd 下所有 deployment 的资源配置 kubectl get deploy -n jxyd -o custom-columns=\ NAME:.metadata.name,\ CPU_LIMIT:.spec.template.spec.containers[0].resources.limits.cpu,\ MEM_LIMIT:.spec.template.spec.containers[0].resources.limits.memory,\ CPU_REQ:.spec.template.spec.containers[0].resources.requests.cpu,\ MEM_REQ:.spec.template.spec.containers[0].resources.requests.memory # 2. 查看某个 deployment 的完整资源配置 kubectl get deploy -n jxyd -o jsonpath='{.spec.template.spec.containers[0].resources}' | python3 -m json.tool # 3. 手动 patch 单个 deployment 的资源 kubectl patch deploy -n jxyd --type='strategic' -p '{ "spec": { "template": { "spec": { "containers": [{ "name": "", "resources": { "limits": {"cpu": "2", "memory": "2Gi"}, "requests": {"cpu": "1", "memory": "500Mi"} }, "env": [{ "name": "CUST_JAVA_OPTS", "value": "-Xms500m -Xmx2000m -Dlog4j2.formatMsgNoLookups=true" }] }] } } } }' # 4. 查看哪些 Pod 在待清退节点上 for node in 10.20.1.142 10.20.1.144 10.20.1.145; do echo "=== $node ===" kubectl get pods -n jxyd --field-selector spec.nodeName=$node -o wide done # 5. 单独给 StatefulSet 设置 nodeSelector kubectl patch statefulset -n jxyd --type='strategic' -p '{"spec":{"template":{"spec":{"nodeSelector":{"jxyd-middleware":"true"}}}}}' # 5.1 单独给业务 Deployment 设置 nodeSelector kubectl patch deploy -n jxyd --type='strategic' -p '{"spec":{"template":{"spec":{"nodeSelector":{"jxyd-business":"true"}}}}}' # 6. 快速 cordon（标记不可调度） kubectl cordon 10.20.1.142 kubectl cordon 10.20.1.144 kubectl cordon 10.20.1.145 # 7. 快速 drain（驱逐 + 清空） kubectl drain 10.20.1.142 --ignore-daemonsets --delete-emptydir-data --force --grace-period=60 kubectl drain 10.20.1.144 --ignore-daemonsets --delete-emptydir-data --force --grace-period=60 kubectl drain 10.20.1.145 --ignore-daemonsets --delete-emptydir-data --force --grace-period=60 ``` --- ## 八、注意事项与风险提示 ### ⚠️ 关键风险点 | 风险 | 说明 | 应对措施 | |------|------|----------| | 中间件识别遗漏 | 自动识别依赖关键词，可能漏判 | **先用 `--list` 模式人工审核** | | PVC/PV 数据丢失 | StatefulSet 的持久化数据可能绑定在特定节点的 local PV | 提前检查 PV 类型，如为 `local-storage` 需手动迁移数据 | | 资源降配影响服务 | CPU/内存缩减可能导致 OOM 或性能下降 | 降配后监控1-2天，关注 Pod 重启情况 | | Strategic Merge Patch 覆盖 env | patch containers 时注意 env 的合并策略 | 脚本已处理，但建议 patch 后验证 | | Drain 超时 | 存在强制终止 Pod 的风险 | 已设置 `grace-period=60` 和 `timeout=300s` | ### ✅ 执行顺序清单 ``` □ Phase 1: 执行备份脚本，确认备份完整 □ Phase 2: 先 --dry-run 预览资源调整 □ Phase 2: --apply 执行资源调整（仅针对 Deployment 业务） □ Phase 2: 监控1-2天确认服务稳定 □ Phase 3: 给保留节点打 label □ Phase 3: 先 --list 查看迁移和调度计划（并确认 hostPath） □ Phase 3: 人工确认中间件识别结果，如有遗漏则修改 MIDDLEWARE_KEYWORDS □ Phase 3: --dry-run 预览迁移与调度 □ Phase 3: --apply 执行迁移与调度 □ Phase 3: --verify 验证迁移与调度结果 □ Phase 3: 监控中间件服务是否正常 □ ---- 等待一个月 ---- □ Phase 4: 执行最终清退脚本 □ Phase 4: 验证集群状态正常 ```