# 节点驱逐 - 完整解决方案

## 一、环境概览

### 集群节点信息

| 节点 IP       | 角色                          | 操作系统                                | 状态   | 处理方式               |
|---------------|-------------------------------|-----------------------------------------|--------|------------------------|
| 10.20.1.130   | controlplane, etcd, worker    | openEuler 20.03 (LTS-SP3)              | 保留   | master 节点不动        |
| 10.20.1.133   | worker                        | openEuler 20.03 (LTS-SP3)              | 保留   | 中间件调度目标         |
| 10.20.1.134   | worker                        | openEuler 20.03 (LTS-SP3)              | 保留   | 中间件调度目标 (mysql) |
| 10.20.1.141   | worker                        | BigCloud Enterprise Linux For Euler     | 保留   | 中间件调度目标         |
| **10.20.1.142** | worker                      | BigCloud Enterprise Linux For Euler     | **清退** | 一个月后清退           |
| **10.20.1.144** | worker                      | BigCloud Enterprise Linux For Euler     | **清退** | 一个月后清退           |
| **10.20.1.145** | worker                      | BigCloud Enterprise Linux For Euler     | **清退** | 一个月后清退           |

### 需求拆解

1. **清退节点**：10.20.1.142、10.20.1.144、10.20.1.145（一个月后执行）
2. **jxyd 命名空间 Deployment**：生命周期与被清退节点一致 → 即**直接删除**
3. **jxyd 命名空间中间件**：需保留，迁移调度到 10.20.1.133、10.20.1.134、10.20.1.141
4. **资源调整**：超标 Deployment 统一降配 + 同步调整 JVM 参数

---

## 二、操作流程总览

```
┌─────────────────────────────────────────────────────────────────┐
│  Phase 1: 信息采集与备份（立即执行）                            │
│  - 导出所有 deployment/statefulset 信息                        │
│  - 记录当前 Pod 分布                                           │
│  - 备份所有 YAML                                               │
├─────────────────────────────────────────────────────────────────┤
│  Phase 2: 资源调整（立即执行）                                  │
│  - 扫描 jxyd 业务 deployment 的资源配置（不扫描中间件）         │
│  - 将超标资源统一降配                                          │
│  - 同步修改 CUST_JAVA_OPTS 环境变量                            │
├─────────────────────────────────────────────────────────────────┤
│  Phase 3: 中间件及业务迁移（立即执行）                          │
│  - 给保留节点及待清退节点打 label                              │
│  - 调度中间件至保留节点，调度业务 Deployment 至待清退节点       │
│  - 验证迁移及调度在对应节点上正常运行                           │
├─────────────────────────────────────────────────────────────────┤
│  Phase 4: 节点清退（一个月后执行）                              │
│  - 删除 jxyd 命名空间中的业务 deployment                       │
│  - cordon + drain 节点                                         │
│  - 从集群中移除节点                                            │
└─────────────────────────────────────────────────────────────────┘
```

---

## 三、Phase 1：信息采集与备份

### 1.1 备份脚本 `phase1_backup.sh`

```bash
#!/bin/bash
# ============================================================
# Phase 1: 信息采集与备份
# 在 master 节点 (10.20.1.130) 上执行
# ============================================================

BACKUP_DIR="/root/wdd/backup_260617/$(date +%Y%m%d_%H%M%S)"
NAMESPACE="jxyd"
mkdir -p "${BACKUP_DIR}"

echo "=========================================="
echo "  Phase 1: 信息采集与备份"
echo "  备份目录: ${BACKUP_DIR}"
echo "=========================================="

# 1. 导出节点信息
echo "[1/6] 导出节点信息..."
kubectl get nodes -o wide --show-labels > "${BACKUP_DIR}/nodes_info.txt"

# 2. 导出 jxyd 命名空间所有资源
echo "[2/6] 导出 jxyd 命名空间所有资源 YAML..."
kubectl get all -n ${NAMESPACE} -o yaml > "${BACKUP_DIR}/all_resources.yaml"

# 3. 单独导出每个 deployment
echo "[3/6] 导出每个 Deployment YAML..."
mkdir -p "${BACKUP_DIR}/deployments"
for dep in $(kubectl get deployments -n ${NAMESPACE} -o jsonpath='{.items[*].metadata.name}'); do
    kubectl get deployment "${dep}" -n ${NAMESPACE} -o yaml > "${BACKUP_DIR}/deployments/${dep}.yaml"
    echo "  已备份 deployment: ${dep}"
done

# 4. 单独导出每个 statefulset（中间件通常用 statefulset）
echo "[4/6] 导出每个 StatefulSet YAML..."
mkdir -p "${BACKUP_DIR}/statefulsets"
for sts in $(kubectl get statefulsets -n ${NAMESPACE} -o jsonpath='{.items[*].metadata.name}' 2>/dev/null); do
    kubectl get statefulset "${sts}" -n ${NAMESPACE} -o yaml > "${BACKUP_DIR}/statefulsets/${sts}.yaml"
    echo "  已备份 statefulset: ${sts}"
done

# 5. 导出 ConfigMap 和 Secret
echo "[5/6] 导出 ConfigMap 和 Secret..."
mkdir -p "${BACKUP_DIR}/configmaps" "${BACKUP_DIR}/secrets"
for cm in $(kubectl get configmaps -n ${NAMESPACE} -o jsonpath='{.items[*].metadata.name}'); do
    kubectl get configmap "${cm}" -n ${NAMESPACE} -o yaml > "${BACKUP_DIR}/configmaps/${cm}.yaml"
done
for sec in $(kubectl get secrets -n ${NAMESPACE} -o jsonpath='{.items[*].metadata.name}'); do
    kubectl get secret "${sec}" -n ${NAMESPACE} -o yaml > "${BACKUP_DIR}/secrets/${sec}.yaml"
done

# 6. 导出 Pod 分布信息（关键：记录哪些 Pod 在待清退节点上）
echo "[6/6] 导出 Pod 分布信息..."
echo "=== 所有 Pod 的节点分布 ===" > "${BACKUP_DIR}/pod_distribution.txt"
kubectl get pods -n ${NAMESPACE} -o wide >> "${BACKUP_DIR}/pod_distribution.txt"

echo ""
echo "--- 待清退节点上的 Pod ---" >> "${BACKUP_DIR}/pod_distribution.txt"
for NODE in 10.20.1.142 10.20.1.144 10.20.1.145; do
    echo "" >> "${BACKUP_DIR}/pod_distribution.txt"
    echo "=== 节点 ${NODE} 上的 Pod ===" >> "${BACKUP_DIR}/pod_distribution.txt"
    kubectl get pods -n ${NAMESPACE} -o wide --field-selector spec.nodeName=${NODE} >> "${BACKUP_DIR}/pod_distribution.txt"
done

# 7. 导出 Service 和 Ingress
mkdir -p "${BACKUP_DIR}/services" "${BACKUP_DIR}/ingresses"
for svc in $(kubectl get svc -n ${NAMESPACE} -o jsonpath='{.items[*].metadata.name}'); do
    kubectl get svc "${svc}" -n ${NAMESPACE} -o yaml > "${BACKUP_DIR}/services/${svc}.yaml"
done
for ing in $(kubectl get ingress -n ${NAMESPACE} -o jsonpath='{.items[*].metadata.name}' 2>/dev/null); do
    kubectl get ingress "${ing}" -n ${NAMESPACE} -o yaml > "${BACKUP_DIR}/ingresses/${ing}.yaml"
done

# 8. 生成资源使用汇总
echo ""
echo "=== 资源使用汇总 ===" | tee "${BACKUP_DIR}/resource_summary.txt"
kubectl top nodes 2>/dev/null | tee -a "${BACKUP_DIR}/resource_summary.txt"
echo "" | tee -a "${BACKUP_DIR}/resource_summary.txt"
kubectl top pods -n ${NAMESPACE} 2>/dev/null | tee -a "${BACKUP_DIR}/resource_summary.txt"

echo ""
echo "=========================================="
echo "  备份完成！目录: ${BACKUP_DIR}"
echo "=========================================="
```

---

## 四、Phase 2：资源检查与调整

### 2.1 Python 脚本 `phase2_resource_adjust.py`

此脚本会：
- 扫描 jxyd 命名空间下所有 Deployment（中间件不存在资源限制，且只在 StatefulSet，因此阶段二不考虑中间件）
- 检测哪些容器的资源超标（超过 limits cpu:2 / memory:2Gi）
- 对超标的 Deployment 自动执行 `kubectl patch` 降配
- 同步修改 `CUST_JAVA_OPTS` 环境变量

```python
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Phase 2: 资源检查与调整
功能：
  1. 扫描 jxyd 命名空间下所有 Deployment 的资源配置
  2. 识别超标资源（超过 limits cpu:2 / memory:2Gi）
  3. 自动 patch 降配 + 修改 CUST_JAVA_OPTS
  4. 生成变更报告

使用方式：
  python3 phase2_resource_adjust.py --dry-run     # 仅预览，不执行变更
  python3 phase2_resource_adjust.py --apply        # 执行变更
"""

import json
import subprocess
import sys
import re
import argparse
from datetime import datetime


# ============================================================
# 配置区
# ============================================================
NAMESPACE = "jxyd"

# 目标资源规格
TARGET_RESOURCES = {
    "limits": {"cpu": "2", "memory": "2Gi"},
    "requests": {"cpu": "1", "memory": "500Mi"}
}

# 目标 JVM 参数
TARGET_JAVA_OPTS = "-Xms500m -Xmx2000m -Dlog4j2.formatMsgNoLookups=true"

# ============================================================
# 工具函数
# ============================================================

def run_kubectl(args, capture=True):
    """执行 kubectl 命令"""
    cmd = ["kubectl"] + args
    result = subprocess.run(cmd, capture_output=capture, text=True)
    if result.returncode != 0:
        print(f"  [错误] 命令失败: {' '.join(cmd)}")
        print(f"  stderr: {result.stderr}")
        return None
    return result.stdout


def parse_memory(mem_str):
    """将内存字符串转换为字节数"""
    if not mem_str:
        return 0
    mem_str = str(mem_str)
    units = {
        'Ki': 1024, 'Mi': 1024**2, 'Gi': 1024**3, 'Ti': 1024**4,
        'K': 1000, 'M': 1000**2, 'G': 1000**3, 'T': 1000**4,
        'k': 1000, 'm': 0.001  # millibytes (unusual but valid)
    }
    for suffix, multiplier in sorted(units.items(), key=lambda x: -len(x[0])):
        if mem_str.endswith(suffix):
            try:
                return float(mem_str[:-len(suffix)]) * multiplier
            except ValueError:
                return 0
    try:
        return float(mem_str)  # 纯数字，单位为字节
    except ValueError:
        return 0


def parse_cpu(cpu_str):
    """将 CPU 字符串转换为核心数（float）"""
    if not cpu_str:
        return 0.0
    cpu_str = str(cpu_str)
    if cpu_str.endswith('m'):
        return float(cpu_str[:-1]) / 1000.0
    return float(cpu_str)


def is_resource_over_limit(resources):
    """检查资源是否超标"""
    limits = resources.get("limits", {})
    cpu_limit = limits.get("cpu", "0")
    mem_limit = limits.get("memory", "0")
    
    cpu_over = parse_cpu(cpu_limit) > parse_cpu(TARGET_RESOURCES["limits"]["cpu"])
    mem_over = parse_memory(mem_limit) > parse_memory(TARGET_RESOURCES["limits"]["memory"])
    
    return cpu_over or mem_over


# ============================================================
# 主逻辑
# ============================================================

def get_deployments():
    """获取所有 Deployment 信息"""
    output = run_kubectl([
        "get", "deployments", "-n", NAMESPACE,
        "-o", "json"
    ])
    if not output:
        return []
    data = json.loads(output)
    return data.get("items", [])


def analyze_and_patch(workloads, kind, dry_run=True):
    """分析并 patch 超标资源，若副本数为 0 则直接删除"""
    report = {
        "business": [],
        "patched": [],
        "skipped": [],
        "errors": [],
        "deleted": []
    }
    
    for wl in workloads:
        name = wl["metadata"]["name"]
        replicas = wl["spec"].get("replicas", 1)
        
        if replicas == 0:
            print(f"\n  [删除] {kind}/{name} (副本数为 0)")
            if dry_run:
                print("    [DRY-RUN] 跳过删除")
                report["skipped"].append({"name": name, "reason": "replicas=0"})
            else:
                result = run_kubectl([
                    "delete", kind.lower(), name,
                    "-n", NAMESPACE
                ])
                if result is not None:
                    print("    [成功] 已删除")
                    report["deleted"].append(name)
                else:
                    print("    [失败] 删除失败")
                    report["errors"].append({"name": name, "reason": "delete failed"})
            continue
            
        spec = wl["spec"]["template"]["spec"]
        containers = spec.get("containers", [])
        
        category = "business"
        
        for idx, container in enumerate(containers):
            container_name = container.get("name", f"container-{idx}")
            resources = container.get("resources", {})
            limits = resources.get("limits", {})
            requests = resources.get("requests", {})
            
            # 提取当前 JAVA_OPTS
            current_java_opts = ""
            env_list = container.get("env", [])
            for env in env_list:
                if env.get("name") == "CUST_JAVA_OPTS":
                    current_java_opts = env.get("value", "")
                    break
            
            item = {
                "name": name,
                "kind": kind,
                "container": container_name,
                "container_index": idx,
                "category": category,
                "current_limits_cpu": limits.get("cpu", "未设置"),
                "current_limits_memory": limits.get("memory", "未设置"),
                "current_requests_cpu": requests.get("cpu", "未设置"),
                "current_requests_memory": requests.get("memory", "未设置"),
                "current_java_opts": current_java_opts,
                "over_limit": is_resource_over_limit(resources),
            }
            
            report[category].append(item)
            
            # 只调整超标的
            if item["over_limit"]:
                print(f"\n  [超标] {kind}/{name} (容器: {container_name})")
                print(f"    当前: limits={{cpu:{limits.get('cpu', 'N/A')}, memory:{limits.get('memory', 'N/A')}}}")
                print(f"    目标: limits={{cpu:{TARGET_RESOURCES['limits']['cpu']}, memory:{TARGET_RESOURCES['limits']['memory']}}}")
                
                if dry_run:
                    print(f"    [DRY-RUN] 跳过 patch")
                    report["skipped"].append(item)
                else:
                    # 构造 patch JSON
                    patch = {
                        "spec": {
                            "template": {
                                "spec": {
                                    "containers": []
                                }
                            }
                        }
                    }
                    
                    # 构建容器 patch（需要按索引）
                    container_patch = {
                        "name": container_name,
                        "resources": TARGET_RESOURCES
                    }
                    
                    # 同步修改 CUST_JAVA_OPTS（如果存在该 env）
                    if current_java_opts:
                        # 使用 kubectl set env 更简洁，这里用 strategic merge patch
                        new_env = []
                        for env in env_list:
                            if env.get("name") == "CUST_JAVA_OPTS":
                                new_env.append({
                                    "name": "CUST_JAVA_OPTS",
                                    "value": TARGET_JAVA_OPTS
                                })
                            else:
                                new_env.append(env)
                        container_patch["env"] = new_env
                    
                    patch["spec"]["template"]["spec"]["containers"] = [container_patch]
                    
                    patch_json = json.dumps(patch)
                    
                    result = run_kubectl([
                        "patch", kind.lower(), name,
                        "-n", NAMESPACE,
                        "--type", "strategic",
                        "-p", patch_json
                    ])
                    
                    if result is not None:
                        print(f"    [成功] 已 patch {kind}/{name}")
                        report["patched"].append(item)
                    else:
                        print(f"    [失败] patch {kind}/{name} 失败")
                        report["errors"].append(item)
            else:
                # 未超标但可能也需要同步 JAVA_OPTS
                if current_java_opts and current_java_opts != TARGET_JAVA_OPTS:
                    print(f"\n  [JAVA_OPTS 不一致] {kind}/{name} (容器: {container_name})")
                    print(f"    当前: {current_java_opts}")
                    print(f"    目标: {TARGET_JAVA_OPTS}")
                    
                    if not dry_run:
                        result = run_kubectl([
                            "set", "env",
                            f"{kind.lower()}/{name}",
                            "-n", NAMESPACE,
                            f"CUST_JAVA_OPTS={TARGET_JAVA_OPTS}",
                            "-c", container_name
                        ])
                        if result is not None:
                            print(f"    [成功] 已更新 JAVA_OPTS")
                        else:
                            print(f"    [失败] 更新 JAVA_OPTS 失败")
    
    return report


def print_summary(report, kind):
    """打印汇总报告"""
    print(f"\n{'='*60}")
    print(f"  {kind} 资源扫描报告")
    print(f"{'='*60}")
    
    print(f"\n  🚀 业务 Deployment ({len(report['business'])} 个容器):")
    for item in report["business"]:
        over = "⚠️ 超标" if item["over_limit"] else "✅ 正常"
        print(f"    - {item['name']}/{item['container']}: "
              f"limits(cpu={item['current_limits_cpu']}, mem={item['current_limits_memory']}) "
              f"requests(cpu={item['current_requests_cpu']}, mem={item['current_requests_memory']}) "
              f"{over}")
    
    if report.get("deleted"):
        print(f"\n  🗑️ 已删除 (副本数为0): {len(report['deleted'])} 个")
    if report["patched"]:
        print(f"\n  🔧 已 Patch: {len(report['patched'])} 个")
    if report["skipped"]:
        print(f"  ⏭️  跳过 (DRY-RUN): {len(report['skipped'])} 个")
    if report["errors"]:
        print(f"  ❌ 失败: {len(report['errors'])} 个")


def main():
    parser = argparse.ArgumentParser(description="Phase 2: jxyd 命名空间资源检查与调整")
    group = parser.add_mutually_exclusive_group(required=True)
    group.add_argument("--dry-run", action="store_true", help="仅预览，不执行变更")
    group.add_argument("--apply", action="store_true", help="执行变更")
    args = parser.parse_args()
    
    dry_run = args.dry_run
    
    print(f"\n{'#'*60}")
    print(f"  Phase 2: 资源检查与调整")
    print(f"  命名空间: {NAMESPACE}")
    print(f"  模式: {'DRY-RUN（仅预览）' if dry_run else '⚡ APPLY（执行变更）'}")
    print(f"  时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
    print(f"{'#'*60}")
    
    if not dry_run:
        confirm = input("\n  ⚠️  确认要执行资源调整吗？(yes/no): ")
        if confirm.lower() != "yes":
            print("  已取消。")
            return
    
    # 扫描 Deployment (业务)
    print(f"\n{'='*60}")
    print("  扫描 Deployment...")
    print(f"{'='*60}")
    deployments = get_deployments()
    dep_report = analyze_and_patch(deployments, "Deployment", dry_run)
    print_summary(dep_report, "Deployment")
    
    # 生成变更日志
    log_file = f"/root/wdd/backup_260617/phase2_report_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"
    log_data = {
        "timestamp": datetime.now().isoformat(),
        "mode": "dry-run" if dry_run else "apply",
        "deployment_report": {
            "business_count": len(dep_report["business"]),
            "patched_count": len(dep_report["patched"]),
            "error_count": len(dep_report["errors"]),
        }
    }
    
    try:
        with open(log_file, 'w') as f:
            json.dump(log_data, f, indent=2, ensure_ascii=False)
        print(f"\n  📄 报告已保存到: {log_file}")
    except Exception as e:
        print(f"\n  [警告] 无法保存报告文件: {e}")
    
    print(f"\n{'#'*60}")
    print(f"  Phase 2 完成!")
    print(f"{'#'*60}")


if __name__ == "__main__":
    main()
```

---

## 五、Phase 3：中间件迁移

### 3.1 给保留节点打 Label

```bash
#!/bin/bash
# ============================================================
# Phase 3.1: 给保留节点打 Label
# ============================================================

echo "=== 给保留节点打 label: jxyd-middleware=true ==="

# 保留节点
kubectl label node 10.20.1.133 jxyd-middleware=true --overwrite
kubectl label node 10.20.1.134 jxyd-middleware=true --overwrite
kubectl label node 10.20.1.141 jxyd-middleware=true --overwrite

echo ""
echo ""
echo "=== 给待清退节点打 label: jxyd-business=true ==="
# 待清退节点（将业务部署到这些节点）
kubectl label node 10.20.1.142 jxyd-business=true --overwrite
kubectl label node 10.20.1.144 jxyd-business=true --overwrite
kubectl label node 10.20.1.145 jxyd-business=true --overwrite

echo ""
echo "=== 验证 label ==="
kubectl get nodes -l jxyd-middleware=true -o wide
kubectl get nodes -l jxyd-business=true -o wide
```

### 3.2 调度与迁移脚本 `phase3_migrate_workloads.py`

```python
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Phase 3: 工作负载迁移与调度
功能：
  1. 识别 jxyd 命名空间下的中间件 StatefulSet 与 业务 Deployment
  2. 为中间件添加 nodeSelector，调度到保留节点（过滤包含 hostPath 的中间件）
  3. 为业务 Deployment 添加 nodeSelector，调度到待清退节点
  4. 验证调度和迁移结果

使用方式：
  python3 phase3_migrate_workloads.py --list             # 列出迁移计划
  python3 phase3_migrate_workloads.py --dry-run           # 预览变更
  python3 phase3_migrate_workloads.py --apply             # 执行迁移
  python3 phase3_migrate_workloads.py --verify            # 验证迁移结果
"""

import json
import subprocess
import sys
import argparse
from datetime import datetime


NAMESPACE = "jxyd"

# 目标节点 label
TARGET_NODE_SELECTOR = {"jxyd-middleware": "true"}
BUSINESS_NODE_SELECTOR = {"jxyd-business": "true"}

# 中间件关键词
MIDDLEWARE_KEYWORDS = [
    "mysql", "redis", "rabbitmq", "kafka", "zookeeper", "elasticsearch",
    "nacos", "minio", "mongo", "postgres", "nginx",
    "sentinel", "rocketmq", "emqx", "mqtt", "influxdb", "grafana",
    "prometheus", "xxl-job", "seata"
]

# 待清退节点
EVICT_NODES = ["10.20.1.142", "10.20.1.144", "10.20.1.145"]


def run_kubectl(args):
    cmd = ["kubectl"] + args
    result = subprocess.run(cmd, capture_output=True, text=True)
    if result.returncode != 0:
        print(f"  [错误] {' '.join(cmd)}: {result.stderr.strip()}")
        return None
    return result.stdout


def is_middleware(name):
    name_lower = name.lower()
    return any(kw in name_lower for kw in MIDDLEWARE_KEYWORDS)


def get_workloads(kind):
    output = run_kubectl(["get", kind, "-n", NAMESPACE, "-o", "json"])
    if not output:
        return []
    return json.loads(output).get("items", [])


def list_workloads():
    """列出工作负载迁移与调度计划"""
    print(f"\n{'='*60}")
    print(f"  jxyd 命名空间 - 迁移与调度计划")
    print(f"{'='*60}")
    
    # StatefulSet (中间件)
    sts_workloads = get_workloads("statefulsets")
    mw_list = [w for w in sts_workloads if is_middleware(w["metadata"]["name"])]
    print(f"\n  📦 中间件 StatefulSet (调度到保留节点，{len(mw_list)} 个):")
    for w in mw_list:
        name = w["metadata"]["name"]
        replicas = w["spec"].get("replicas", 1)
        node_selector = w["spec"]["template"]["spec"].get("nodeSelector", {})
        volumes = w["spec"]["template"]["spec"].get("volumes", [])
        has_host_path = any("hostPath" in v for v in volumes)
        host_path_flag = "[含 hostPath，不可迁移]" if has_host_path else ""
        print(f"    ✅ {name}  (replicas={replicas}, nodeSelector={node_selector}) {host_path_flag}")

    # Deployment (业务)
    dep_workloads = get_workloads("deployments")
    print(f"\n  🚀 业务 Deployment (调度到待清退节点，{len(dep_workloads)} 个):")
    for w in dep_workloads:
        name = w["metadata"]["name"]
        replicas = w["spec"].get("replicas", 1)
        node_selector = w["spec"]["template"]["spec"].get("nodeSelector", {})
        if replicas == 0:
            print(f"    - {name}  (replicas=0, 忽略调度)")
        else:
            print(f"    ✅ {name}  (replicas={replicas}, nodeSelector={node_selector})")
    
    print(f"\n  💡 提示: 若状态不准确可人工审核配置。")


def migrate_workloads(dry_run=True):
    """执行工作负载的迁移与调度"""
    print(f"\n{'='*60}")
    print(f"  工作负载调度 - {'DRY-RUN' if dry_run else 'APPLY'}")
    print(f"{'='*60}")
    
    patched = 0
    errors = 0
    
    def apply_patch(kind, w, target_ns, is_middleware_check=False):
        nonlocal patched, errors
        name = w["metadata"]["name"]
        replicas = w["spec"].get("replicas", 1)
        current_ns = w["spec"]["template"]["spec"].get("nodeSelector", {})
        
        if replicas == 0:
            return  # 忽略副本数为 0 的
            
        if is_middleware_check:
            volumes = w["spec"]["template"]["spec"].get("volumes", [])
            has_host_path = any("hostPath" in v for v in volumes)
            if has_host_path:
                print(f"  [跳过] {kind}/{name} 包含 hostPath 挂载，不能进行迁移")
                return
                
        # 检查是否已包含所有的目标 selector
        if all(current_ns.get(k) == v for k, v in target_ns.items()):
            print(f"  [跳过] {kind}/{name} 已满足 nodeSelector 目标")
            return
            
        print(f"  [调度] {kind}/{name}")
        print(f"    当前 nodeSelector: {current_ns}")
        print(f"    目标 nodeSelector: {target_ns}")
        
        if dry_run:
            print(f"    [DRY-RUN] 跳过")
            return
            
        merged_selector = {**current_ns, **target_ns}
        patch = {"spec": {"template": {"spec": {"nodeSelector": merged_selector}}}}
        result = run_kubectl([
            "patch", kind, name, "-n", NAMESPACE, "--type", "strategic", "-p", json.dumps(patch)
        ])
        
        if result is not None:
            print(f"    [成功] 已 patch")
            patched += 1
        else:
            print(f"    [失败]")
            errors += 1

    # 处理中间件
    sts_workloads = get_workloads("statefulsets")
    for w in [w for w in sts_workloads if is_middleware(w["metadata"]["name"])]:
        apply_patch("statefulset", w, TARGET_NODE_SELECTOR, True)
        
    # 处理业务
    dep_workloads = get_workloads("deployments")
    for w in dep_workloads:
        apply_patch("deployment", w, BUSINESS_NODE_SELECTOR, False)
        
    print(f"\n  汇总: 已调度 {patched} 个, 失败 {errors} 个")


def verify_migration():
    """验证迁移及调度结果"""
    print(f"\n{'='*60}")
    print(f"  调度结果验证")
    print(f"{'='*60}")
    
    issues = []
    
    def check_workloads(kind, workloads, target_label_key, target_nodes_list, is_middleware_check=False):
        for w in workloads:
            name = w["metadata"]["name"]
            replicas = w["spec"].get("replicas", 1)
            if replicas == 0:
                continue
                
            ns = w["spec"]["template"]["spec"].get("nodeSelector", {})
            if is_middleware_check:
                volumes = w["spec"]["template"]["spec"].get("volumes", [])
                if any("hostPath" in v for v in volumes):
                    continue
                    
            if ns.get(target_label_key) != "true":
                issues.append(f"  ⚠️  {kind}/{name} 未设置 {target_label_key} nodeSelector")
                
            pods_output = run_kubectl(["get", "pods", "-n", NAMESPACE, "-l", f"app={name}", "-o", "json"])
            if not pods_output or json.loads(pods_output).get("items", []) == []:
                pods_output = run_kubectl(["get", "pods", "-n", NAMESPACE, "-o", "json"])
                if pods_output:
                    all_pods = [p for p in json.loads(pods_output).get("items", []) if p["metadata"]["name"].startswith(name)]
                else:
                    all_pods = []
            else:
                all_pods = json.loads(pods_output).get("items", [])
                
            for pod in all_pods:
                pod_name = pod["metadata"]["name"]
                node = pod["spec"].get("nodeName", "unknown")
                phase = pod["status"].get("phase", "Unknown")
                
                # Check target nodes
                if target_nodes_list is not None and node not in target_nodes_list:
                    issues.append(f"  ⚠️  Pod {pod_name} 调度异常，当前节点 {node}，应在目标节点集合 {target_nodes_list} 中")
                # Exclude nodes
                elif target_nodes_list is None and node in EVICT_NODES:
                    issues.append(f"  ⚠️  Pod {pod_name} 仍在待清退节点 {node} 上")
                else:
                    print(f"  ✅ {pod_name} -> 节点 {node} (状态: {phase})")

    # 1. 中间件需在保留节点上（非待清退节点）
    sts_workloads = get_workloads("statefulsets")
    mw_list = [w for w in sts_workloads if is_middleware(w["metadata"]["name"])]
    check_workloads("statefulset", mw_list, "jxyd-middleware", None, True)
    
    # 2. 业务需在待清退节点上
    dep_workloads = get_workloads("deployments")
    check_workloads("deployment", dep_workloads, "jxyd-business", EVICT_NODES, False)

    if issues:
        print(f"\n  ⚠️  发现 {len(issues)} 个问题:")
        for issue in issues:
            print(issue)
    else:
        print(f"\n  ✅ 所有的工作负载已成功调度到对应的目标节点!")


def main():
    parser = argparse.ArgumentParser(description="Phase 3: 工作负载迁移与调度")
    group = parser.add_mutually_exclusive_group(required=True)
    group.add_argument("--list", action="store_true", help="列出迁移计划")
    group.add_argument("--dry-run", action="store_true", help="预览变更")
    group.add_argument("--apply", action="store_true", help="执行迁移")
    group.add_argument("--verify", action="store_true", help="验证迁移结果")
    args = parser.parse_args()
    
    if args.list:
        list_workloads()
    elif args.dry_run:
        migrate_workloads(dry_run=True)
    elif args.apply:
        confirm = input("  ⚠️  确认要执行迁移和调度吗？(yes/no): ")
        if confirm.lower() == "yes":
            migrate_workloads(dry_run=False)
        else:
            print("  已取消。")
    elif args.verify:
        verify_migration()


if __name__ == "__main__":
    main()
```

---

## 六、Phase 4：节点清退（一个月后执行）

### 4.1 清退脚本 `phase4_evict_nodes.sh`

```bash
#!/bin/bash
# ============================================================
# Phase 4: 节点清退（一个月后执行）
# 
# ⚠️  此脚本将执行不可逆操作，请确认：
#   1. Phase 2 资源调整已完成
#   2. Phase 3 中间件迁移已完成并验证
#   3. 已获得相关审批
# ============================================================

set -e

NAMESPACE="jxyd"
EVICT_NODES=("10.20.1.142" "10.20.1.144" "10.20.1.145")
LOG_DIR="/root/wdd/backup_260617/phase4_$(date +%Y%m%d_%H%M%S)"
mkdir -p "${LOG_DIR}"

echo "=========================================="
echo "  Phase 4: 节点清退"
echo "  时间: $(date '+%Y-%m-%d %H:%M:%S')"
echo "  日志: ${LOG_DIR}"
echo "=========================================="

# ============================================================
# Step 1: 最终确认
# ============================================================
echo ""
echo "⚠️  即将执行以下操作："
echo "  1. 删除 ${NAMESPACE} 命名空间中的业务 Deployment"
echo "  2. Cordon 节点: ${EVICT_NODES[*]}"
echo "  3. Drain 节点: ${EVICT_NODES[*]}"
echo "  4. 从集群中删除节点: ${EVICT_NODES[*]}"
echo ""
read -p "确认执行？(输入 YES 继续): " CONFIRM
if [ "${CONFIRM}" != "YES" ]; then
    echo "已取消。"
    exit 0
fi

# ============================================================
# Step 2: 最终备份
# ============================================================
echo ""
echo "=== Step 2: 最终备份 ==="
kubectl get all -n ${NAMESPACE} -o yaml > "${LOG_DIR}/final_backup_all.yaml"
kubectl get pods -n ${NAMESPACE} -o wide > "${LOG_DIR}/final_pod_distribution.txt"
echo "  备份完成"

# ============================================================
# Step 3: 删除 jxyd 业务 Deployment
# （说明：中间件仅存在于 StatefulSet，因此所有的 Deployment 均为业务）
# ============================================================
echo ""
echo "=== Step 3: 删除 jxyd 业务 Deployment ==="

# 获取所有 deployment
ALL_DEPS=$(kubectl get deployments -n ${NAMESPACE} -o jsonpath='{.items[*].metadata.name}')

for dep in ${ALL_DEPS}; do
    echo "  [删除-业务] ${dep}"
    kubectl delete deployment "${dep}" -n ${NAMESPACE} --grace-period=30 2>&1 | tee -a "${LOG_DIR}/delete_deployments.log"
done

# ============================================================
# Step 4: Cordon 节点（标记不可调度）
# ============================================================
echo ""
echo "=== Step 4: Cordon 节点 ==="
for NODE in "${EVICT_NODES[@]}"; do
    echo "  Cordon ${NODE}..."
    kubectl cordon "${NODE}" 2>&1 | tee -a "${LOG_DIR}/cordon.log"
done

echo ""
kubectl get nodes -o wide | tee "${LOG_DIR}/nodes_after_cordon.txt"

# ============================================================
# Step 5: Drain 节点（驱逐所有 Pod）
# ============================================================
echo ""
echo "=== Step 5: Drain 节点 ==="
for NODE in "${EVICT_NODES[@]}"; do
    echo "  Drain ${NODE}..."
    kubectl drain "${NODE}" \
        --ignore-daemonsets \
        --delete-emptydir-data \
        --force \
        --grace-period=60 \
        --timeout=300s \
        2>&1 | tee -a "${LOG_DIR}/drain_${NODE}.log"
    
    echo "  等待 30 秒让 Pod 完成迁移..."
    sleep 30
done

# ============================================================
# Step 6: 验证 Pod 已全部迁走
# ============================================================
echo ""
echo "=== Step 6: 验证 Pod 迁移情况 ==="
for NODE in "${EVICT_NODES[@]}"; do
    REMAINING=$(kubectl get pods --all-namespaces --field-selector spec.nodeName=${NODE} --no-headers 2>/dev/null | grep -v "kube-system" | wc -l)
    if [ "${REMAINING}" -gt 0 ]; then
        echo "  ⚠️  节点 ${NODE} 上仍有 ${REMAINING} 个非系统 Pod:"
        kubectl get pods --all-namespaces --field-selector spec.nodeName=${NODE} -o wide | grep -v "kube-system"
    else
        echo "  ✅ 节点 ${NODE} 上的用户 Pod 已全部迁走"
    fi
done

# ============================================================
# Step 7: 从集群中删除节点
# ============================================================
echo ""
echo "=== Step 7: 从集群中删除节点 ==="
read -p "确认从集群中删除节点？(输入 YES 继续): " CONFIRM2
if [ "${CONFIRM2}" == "YES" ]; then
    for NODE in "${EVICT_NODES[@]}"; do
        echo "  删除节点 ${NODE}..."
        kubectl delete node "${NODE}" 2>&1 | tee -a "${LOG_DIR}/delete_nodes.log"
    done
else
    echo "  跳过节点删除。可稍后手动执行: kubectl delete node <node>"
fi

# ============================================================
# Step 8: 最终验证
# ============================================================
echo ""
echo "=== Step 8: 最终验证 ==="
echo "--- 集群节点状态 ---"
kubectl get nodes -o wide | tee "${LOG_DIR}/nodes_final.txt"
echo ""
echo "--- jxyd 命名空间 Pod 状态 ---"
kubectl get pods -n ${NAMESPACE} -o wide | tee "${LOG_DIR}/pods_final.txt"

echo ""
echo "=========================================="
echo "  Phase 4 完成！"
echo "  日志目录: ${LOG_DIR}"
echo "=========================================="
```

---

## 七、快速命令参考

### 如果需要手动单独操作

```bash
# ============================================================
# 常用单条命令（按需执行）
# ============================================================

# 1. 查看 jxyd 下所有 deployment 的资源配置
kubectl get deploy -n jxyd -o custom-columns=\
NAME:.metadata.name,\
CPU_LIMIT:.spec.template.spec.containers[0].resources.limits.cpu,\
MEM_LIMIT:.spec.template.spec.containers[0].resources.limits.memory,\
CPU_REQ:.spec.template.spec.containers[0].resources.requests.cpu,\
MEM_REQ:.spec.template.spec.containers[0].resources.requests.memory

# 2. 查看某个 deployment 的完整资源配置
kubectl get deploy <NAME> -n jxyd -o jsonpath='{.spec.template.spec.containers[0].resources}' | python3 -m json.tool

# 3. 手动 patch 单个 deployment 的资源
kubectl patch deploy <NAME> -n jxyd --type='strategic' -p '{
  "spec": {
    "template": {
      "spec": {
        "containers": [{
          "name": "<CONTAINER_NAME>",
          "resources": {
            "limits": {"cpu": "2", "memory": "2Gi"},
            "requests": {"cpu": "1", "memory": "500Mi"}
          },
          "env": [{
            "name": "CUST_JAVA_OPTS",
            "value": "-Xms500m -Xmx2000m -Dlog4j2.formatMsgNoLookups=true"
          }]
        }]
      }
    }
  }
}'

# 4. 查看哪些 Pod 在待清退节点上
for node in 10.20.1.142 10.20.1.144 10.20.1.145; do
    echo "=== $node ==="
    kubectl get pods -n jxyd --field-selector spec.nodeName=$node -o wide
done

# 5. 单独给 StatefulSet 设置 nodeSelector
kubectl patch statefulset <NAME> -n jxyd --type='strategic' -p '{"spec":{"template":{"spec":{"nodeSelector":{"jxyd-middleware":"true"}}}}}'

# 5.1 单独给业务 Deployment 设置 nodeSelector
kubectl patch deploy <NAME> -n jxyd --type='strategic' -p '{"spec":{"template":{"spec":{"nodeSelector":{"jxyd-business":"true"}}}}}'

# 6. 快速 cordon（标记不可调度）
kubectl cordon 10.20.1.142
kubectl cordon 10.20.1.144
kubectl cordon 10.20.1.145

# 7. 快速 drain（驱逐 + 清空）
kubectl drain 10.20.1.142 --ignore-daemonsets --delete-emptydir-data --force --grace-period=60
kubectl drain 10.20.1.144 --ignore-daemonsets --delete-emptydir-data --force --grace-period=60
kubectl drain 10.20.1.145 --ignore-daemonsets --delete-emptydir-data --force --grace-period=60
```

---

## 八、注意事项与风险提示

### ⚠️  关键风险点

| 风险 | 说明 | 应对措施 |
|------|------|----------|
| 中间件识别遗漏 | 自动识别依赖关键词，可能漏判 | **先用 `--list` 模式人工审核** |
| PVC/PV 数据丢失 | StatefulSet 的持久化数据可能绑定在特定节点的 local PV | 提前检查 PV 类型，如为 `local-storage` 需手动迁移数据 |
| 资源降配影响服务 | CPU/内存缩减可能导致 OOM 或性能下降 | 降配后监控1-2天，关注 Pod 重启情况 |
| Strategic Merge Patch 覆盖 env | patch containers 时注意 env 的合并策略 | 脚本已处理，但建议 patch 后验证 |
| Drain 超时 | 存在强制终止 Pod 的风险 | 已设置 `grace-period=60` 和 `timeout=300s` |

### ✅ 执行顺序清单

```
□ Phase 1: 执行备份脚本，确认备份完整
□ Phase 2: 先 --dry-run 预览资源调整
□ Phase 2: --apply 执行资源调整（仅针对 Deployment 业务）
□ Phase 2: 监控1-2天确认服务稳定
□ Phase 3: 给保留节点打 label
□ Phase 3: 先 --list 查看迁移和调度计划（并确认 hostPath）
□ Phase 3: 人工确认中间件识别结果，如有遗漏则修改 MIDDLEWARE_KEYWORDS
□ Phase 3: --dry-run 预览迁移与调度
□ Phase 3: --apply 执行迁移与调度
□ Phase 3: --verify 验证迁移与调度结果
□ Phase 3: 监控中间件服务是否正常
□ ---- 等待一个月 ----
□ Phase 4: 执行最终清退脚本
□ Phase 4: 验证集群状态正常
```
