TSL是什么

image-20241017141303330

K8s中的作用是什么?

在 Kubernetes(K8s)中,TSL 指的是 Transport Layer Security,也就是传输层安全协议。它是用来保护在网络上传输的数据的安全性和隐私性。

TSL 在 Kubernetes 中的作用包括:

  1. 加密通信

    • TSL 确保 Kubernetes 各个组件(如 API Server、etcd、Kubelet 等)之间的通信是加密的,防止未经授权的用户截获敏感信息。
  2. 身份验证

    • TSL 证书被用来对客户端和服务器进行身份验证。Kubernetes 的 API Server 通常会使用 TSL 证书来验证客户端(如 kubectl 或其他服务)的身份,确保只有经过认证的用户或服务才能访问集群资源。
  3. 数据完整性

    • TSL 确保数据在传输过程中不会被篡改。通过加密哈希技术,它可以验证接收到的数据是否与发送时保持一致,防止攻击者在传输过程中修改数据。
  4. 支持 HTTPS

    • Kubernetes API Server 使用 TSL 来支持安全的 HTTPS 连接,而不是不安全的 HTTP。这样确保管理员、开发人员或其他组件通过 API 访问 Kubernetes 时的数据安全性。

总结一下就是:通过使用 TSL,Kubernetes 可以确保集群内的通信安全,防止常见的网络攻击如中间人攻击(MITM)。

K8s中TSL证书的具体使用形式

  1. CA证书(Certificate Authority Certificate):这是用于签发其他证书的根证书。它通常由集群管理员生成,用于签署 Kubernetes 集群中的各类证书,包括 API Server、kubelet 等组件。CA 证书的有效期一般较长(通常为 10 年),主要是因为更换 CA 证书的影响较大,涉及到重新签发和更新集群中所有其他证书的过程,维护成本较高。

  2. API Server证书:API Server 是 Kubernetes 集群中最核心的组件之一,它与客户端(如 kubectl)、其他控制组件(如 kube-scheduler 和 kube-controller-manager),以及集群节点通信时需要使用证书进行身份验证。API Server 证书的默认有效期通常为 1 年,主要是出于以下几个考虑:

    • 安全性:证书的有效期越短,定期轮换越频繁,能够减少密钥泄露或加密算法被攻破的风险。API Server 作为集群中重要的组件,通信和身份验证的安全性极为重要,因此缩短证书有效期能够减少潜在的安全风险。
    • 自动化更新:在现代 Kubernetes 集群中,已经引入了自动证书轮换机制。Kubernetes 会在证书接近过期时自动重新生成并更新证书,因此一年期的证书不会对集群的日常运行造成太大影响。(这块具体对版本的要求是1.8+,具体的详情可以查看一下官网的介绍:https://kubernetes.io/blog/2017/09/kubernetes-18-security-workloads-and/)

image-20241017152926639

K8s中TSL证书如何获取的

在 Kubernetes (K8s) 集群中,证书的生成方式取决于你使用的安装方法和具体的需求。Kubernetes 中的证书主要用于确保集群组件之间的安全通信。通常有两种方式生成和管理证书:自动生成手动生成

1. 自动生成证书

  • Kubeadm安装:如果你使用 kubeadm 进行集群安装,Kubernetes 会自动生成和管理集群组件所需的证书。在集群初始化时,kubeadm 会自动创建一个自签名的 CA(Certificate Authority,证书颁发机构)证书,并基于该 CA 生成 API Server、kubelet、etcd 等组件所需的其他证书。这些证书的默认有效期是预定义的,比如:

    • CA 证书的有效期通常为 10 年
    • 集群组件(如 API Server)的证书默认有效期为 1 年
  • 证书自动轮换:在现代 Kubernetes 版本中,集群组件的证书支持自动轮换。例如,API Server 和其他组件的证书接近过期时,Kubernetes 会自动更新证书,无需手动干预。

2. 手动申请和管理证书

  • 自定义证书:如果你的组织有更严格的安全要求或需要使用外部的证书颁发机构(CA)签发的证书,你可以手动生成和管理 Kubernetes 集群所需的证书。例如,使用 opensslcfssl 来生成你自己的证书并配置它们。
    • 在这种情况下,你需要确保每个 Kubernetes 组件(如 API Server、Controller Manager、Scheduler、kubelet、etcd)都配置好你手动申请的证书。
    • 手动申请证书的过程需要你手动管理证书的颁发、续期和更新,因此维护起来较为复杂。

3. 使用外部 CA

  • 你可以使用组织内部的证书颁发机构(如企业内部的 PKI 基础设施)或公开的 CA 来签发 Kubernetes 组件所需的证书。在这种情况下,你需要将 Kubernetes 中的组件配置为使用由该外部 CA 签发的证书,而不是使用 kubeadm 自动生成的证书。

如何进行续期

查看证书的过期时间

ca证书有效期
openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -text  |grep Not

image-20241017153216317

apiserver证书有效期
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text  |grep Not

image-20241017153303228

如何延长过期时间

**update-kubeadm-cert.sh**文件上传到K8s的Master节点

脚本的下载地址:https://github.com/yuyicai/update-kube-cert

脚本命令

#!/usr/bin/env bash

set -o errexit
set -o pipefail
# set -o xtrace

# set output color
NC='\033[0m'
RED='\033[31m'
GREEN='\033[32m'
YELLOW='\033[33m'
BLUE='\033[34m'
# set default cri
CRI="docker"

log::err() {
  printf "[$(date +'%Y-%m-%dT%H:%M:%S.%2N%z')][${RED}ERROR${NC}] %b\n" "$@"
}

log::info() {
  printf "[$(date +'%Y-%m-%dT%H:%M:%S.%2N%z')][INFO] %b\n" "$@"
}

log::warning() {
  printf "[$(date +'%Y-%m-%dT%H:%M:%S.%2N%z')][${YELLOW}WARNING${NC}] \033[0m%b\n" "$@"
}

check_file() {
  if [[ ! -r ${1} ]]; then
    log::err "can not find ${1}"
    exit 1
  fi
}

# get x509v3 subject alternative name from the old certificate
cert::get_subject_alt_name() {
  local cert=${1}.crt
  local alt_name

  check_file "${cert}"
  alt_name=$(openssl x509 -text -noout -in "${cert}" | grep -A1 'Alternative' | tail -n1 | sed 's/[[:space:]]*Address//g')
  printf "%s\n" "${alt_name}"
}

# get subject from the old certificate
cert::get_subj() {
  local cert=${1}.crt
  local subj

  check_file "${cert}"
  subj=$(openssl x509 -text -noout -in "${cert}" | grep "Subject:" | sed 's/Subject:/\//g;s/\,/\//;s/[[:space:]]//g')
  printf "%s\n" "${subj}"
}

cert::backup_file() {
  local file=${1}
  if [[ ! -e ${file}.old-$(date +%Y%m%d) ]]; then
    cp -rp "${file}" "${file}.old-$(date +%Y%m%d)"
    log::info "backup ${file} to ${file}.old-$(date +%Y%m%d)"
  else
    log::warning "does not backup, ${file}.old-$(date +%Y%m%d) already exists"
  fi
}

# check certificate expiration
cert::check_cert_expiration() {
  local cert=${1}.crt
  local cert_expires

  cert_expires=$(openssl x509 -text -noout -in "${cert}" | awk -F ": " '/Not After/{print$2}')
  printf "%s\n" "${cert_expires}"
}

# check kubeconfig expiration
cert::check_kubeconfig_expiration() {
  local config=${1}.conf
  local cert
  local cert_expires

  cert=$(grep "client-certificate-data" "${config}" | awk '{print$2}' | base64 -d)
  cert_expires=$(openssl x509 -text -noout -in <(printf "%s" "${cert}") | awk -F ": " '/Not After/{print$2}')
  printf "%s\n" "${cert_expires}"
}

# check etcd certificates expiration
cert::check_etcd_certs_expiration() {
  local cert
  local certs

  certs=(
    "${ETCD_CERT_CA}"
    "${ETCD_CERT_SERVER}"
    "${ETCD_CERT_PEER}"
    "${ETCD_CERT_HEALTHCHECK_CLIENT}"
    "${ETCD_CERT_APISERVER_ETCD_CLIENT}"
  )

  for cert in "${certs[@]}"; do
    if [[ ! -r ${cert} ]]; then
      printf "%-50s%-30s\n" "${cert}.crt" "$(cert::check_cert_expiration "${cert}")"
    fi
  done
}

# check master certificates expiration
cert::check_master_certs_expiration() {
  local certs
  local kubeconfs
  local cert
  local conf

  certs=(
    "${CERT_CA}"
    "${CERT_APISERVER}"
    "${CERT_APISERVER_KUBELET_CLIENT}"
    "${FRONT_PROXY_CA}"
    "${FRONT_PROXY_CLIENT}"
  )

  # add support for super_admin.conf, which was added after k8s v1.30.
  if [ -f "${CONF_SUPER_ADMIN}.conf" ]; then
    kubeconfs=(
      "${CONF_CONTROLLER_MANAGER}"
      "${CONF_SCHEDULER}"
      "${CONF_ADMIN}"
      "${CONF_SUPER_ADMIN}"
    )
  else 
    kubeconfs=(
      "${CONF_CONTROLLER_MANAGER}"
      "${CONF_SCHEDULER}"
      "${CONF_ADMIN}"
    )
  fi

  printf "%-50s%-30s\n" "CERTIFICATE" "EXPIRES"

  for conf in "${kubeconfs[@]}"; do
    if [[ ! -r ${conf} ]]; then
      printf "%-50s%-30s\n" "${conf}.config" "$(cert::check_kubeconfig_expiration "${conf}")"
    fi
  done

  for cert in "${certs[@]}"; do
    if [[ ! -r ${cert} ]]; then
      printf "%-50s%-30s\n" "${cert}.crt" "$(cert::check_cert_expiration "${cert}")"
    fi
  done
}

# check all certificates expiration
cert::check_all_expiration() {
  cert::check_master_certs_expiration
  cert::check_etcd_certs_expiration
}

# generate certificate whit client, server or peer
# Args:
#   $1 (the name of certificate)
#   $2 (the type of certificate, must be one of client, server, peer)
#   $3 (the subject of certificates)
#   $4 (the validity of certificates) (days)
#   $5 (the name of ca)
#   $6 (the x509v3 subject alternative name of certificate when the type of certificate is server or peer)
cert::gen_cert() {
  local cert_name=${1}
  local cert_type=${2}
  local subj=${3}
  local cert_days=${4}
  local ca_name=${5}
  local alt_name=${6}
  local ca_cert=${ca_name}.crt
  local ca_key=${ca_name}.key
  local cert=${cert_name}.crt
  local key=${cert_name}.key
  local csr=${cert_name}.csr
  local common_csr_conf='distinguished_name = dn\n[dn]\n[v3_ext]\nkeyUsage = critical, digitalSignature, keyEncipherment\n'

  for file in "${ca_cert}" "${ca_key}" "${cert}" "${key}"; do
    check_file "${file}"
  done

  case "${cert_type}" in
  client)
    csr_conf=$(printf "%bextendedKeyUsage = clientAuth\n" "${common_csr_conf}")
    ;;
  server)
    csr_conf=$(printf "%bextendedKeyUsage = serverAuth\nsubjectAltName = %b\n" "${common_csr_conf}" "${alt_name}")
    ;;
  peer)
    csr_conf=$(printf "%bextendedKeyUsage = serverAuth, clientAuth\nsubjectAltName = %b\n" "${common_csr_conf}" "${alt_name}")
    ;;
  *)
    log::err "unknow, unsupported certs type: ${YELLOW}${cert_type}${NC}, supported type: client, server, peer"
    exit 1
    ;;
  esac

  # gen csr
  openssl req -new -key "${key}" -subj "${subj}" -reqexts v3_ext \
    -config <(printf "%b" "${csr_conf}") \
    -out "${csr}" >/dev/null 2>&1
  # gen cert
  openssl x509 -in "${csr}" -req -CA "${ca_cert}" -CAkey "${ca_key}" -CAcreateserial -extensions v3_ext \
    -extfile <(printf "%b" "${csr_conf}") \
    -days "${cert_days}" -out "${cert}" >/dev/null 2>&1

  rm -f "${csr}"
}

cert::update_kubeconf() {
  local cert_name=${1}
  local kubeconf_file=${cert_name}.conf
  local cert=${cert_name}.crt
  local key=${cert_name}.key
  local subj
  local cert_base64

  check_file "${kubeconf_file}"
  # get the key from the old kubeconf
  grep "client-key-data" "${kubeconf_file}" | awk '{print$2}' | base64 -d >"${key}"
  # get the old certificate from the old kubeconf
  grep "client-certificate-data" "${kubeconf_file}" | awk '{print$2}' | base64 -d >"${cert}"
  # get subject from the old certificate
  subj=$(cert::get_subj "${cert_name}")
  cert::gen_cert "${cert_name}" "client" "${subj}" "${CERT_DAYS}" "${CERT_CA}"
  # get certificate base64 code
  cert_base64=$(base64 -w 0 "${cert}")

  # set certificate base64 code to kubeconf
  sed -i 's/client-certificate-data:.*/client-certificate-data: '"${cert_base64}"'/g' "${kubeconf_file}"

  rm -f "${cert}"
  rm -f "${key}"
}

cert::update_etcd_cert() {
  local subj
  local subject_alt_name
  local cert

  # generate etcd server,peer certificate
  # /etc/kubernetes/pki/etcd/server
  # /etc/kubernetes/pki/etcd/peer
  for cert in ${ETCD_CERT_SERVER} ${ETCD_CERT_PEER}; do
    subj=$(cert::get_subj "${cert}")
    subject_alt_name=$(cert::get_subject_alt_name "${cert}")
    cert::gen_cert "${cert}" "peer" "${subj}" "${CERT_DAYS}" "${ETCD_CERT_CA}" "${subject_alt_name}"
    log::info "${GREEN}updated ${BLUE}${cert}.conf${NC}"
  done

  # generate etcd healthcheck-client,apiserver-etcd-client certificate
  # /etc/kubernetes/pki/etcd/healthcheck-client
  # /etc/kubernetes/pki/apiserver-etcd-client
  for cert in ${ETCD_CERT_HEALTHCHECK_CLIENT} ${ETCD_CERT_APISERVER_ETCD_CLIENT}; do
    subj=$(cert::get_subj "${cert}")
    cert::gen_cert "${cert}" "client" "${subj}" "${CERT_DAYS}" "${ETCD_CERT_CA}"
    log::info "${GREEN}updated ${BLUE}${cert}.conf${NC}"
  done

  # restart etcd
  case $CRI in
    "docker")
      docker ps | awk '/k8s_etcd/{print$1}' | xargs -r -I '{}' docker restart {} >/dev/null 2>&1 || true
      ;;
    "containerd")
      crictl ps | awk '/etcd-/{print$(NF-1)}' | xargs -r -I '{}' crictl stopp {} >/dev/null 2>&1 || true
      ;;
  esac
  log::info "restarted etcd with ${CRI}"
}

cert::update_master_cert() {
  local subj
  local subject_alt_name
  local conf

  # generate apiserver server certificate
  # /etc/kubernetes/pki/apiserver
  subj=$(cert::get_subj "${CERT_APISERVER}")
  subject_alt_name=$(cert::get_subject_alt_name "${CERT_APISERVER}")
  cert::gen_cert "${CERT_APISERVER}" "server" "${subj}" "${CERT_DAYS}" "${CERT_CA}" "${subject_alt_name}"
  log::info "${GREEN}updated ${BLUE}${CERT_APISERVER}.crt${NC}"

  # generate apiserver-kubelet-client certificate
  # /etc/kubernetes/pki/apiserver-kubelet-client
  subj=$(cert::get_subj "${CERT_APISERVER_KUBELET_CLIENT}")
  cert::gen_cert "${CERT_APISERVER_KUBELET_CLIENT}" "client" "${subj}" "${CERT_DAYS}" "${CERT_CA}"
  log::info "${GREEN}updated ${BLUE}${CERT_APISERVER_KUBELET_CLIENT}.crt${NC}"

  # generate kubeconf for controller-manager,scheduler and kubelet
  # /etc/kubernetes/controller-manager,scheduler,admin,kubelet.conf,super_admin(added after k8s v1.30.)

  if [ -f "${CONF_SUPER_ADMIN}.conf" ]; then
    conf_list="${CONF_CONTROLLER_MANAGER} ${CONF_SCHEDULER} ${CONF_ADMIN} ${CONF_KUBELET} ${CONF_SUPER_ADMIN}"
  else 
    conf_list="${CONF_CONTROLLER_MANAGER} ${CONF_SCHEDULER} ${CONF_ADMIN} ${CONF_KUBELET}"
  fi
  
  for conf in ${conf_list}; do
    if [[ ${conf##*/} == "kubelet" ]]; then
      # https://github.com/kubernetes/kubeadm/issues/1753
      set +e
      grep kubelet-client-current.pem /etc/kubernetes/kubelet.conf >/dev/null 2>&1
      kubelet_cert_auto_update=$?
      set -e
      if [[ "$kubelet_cert_auto_update" == "0" ]]; then
        log::info "does not need to update kubelet.conf"
        continue
      fi
    fi

    # update kubeconf
    cert::update_kubeconf "${conf}"
    log::info "${GREEN}updated ${BLUE}${conf}.conf${NC}"

    # copy admin.conf to ${HOME}/.kube/config
    if [[ ${conf##*/} == "admin" ]]; then
      mkdir -p "${HOME}/.kube"
      local config=${HOME}/.kube/config
      local config_backup
      config_backup=${HOME}/.kube/config.old-$(date +%Y%m%d)
      if [[ -f ${config} ]] && [[ ! -f ${config_backup} ]]; then
        cp -fp "${config}" "${config_backup}"
        log::info "backup ${config} to ${config_backup}"
      fi
      cp -fp "${conf}.conf" "${HOME}/.kube/config"
      log::info "copy the admin.conf to ${HOME}/.kube/config"
    fi
  done

  # generate front-proxy-client certificate
  # /etc/kubernetes/pki/front-proxy-client
  subj=$(cert::get_subj "${FRONT_PROXY_CLIENT}")
  cert::gen_cert "${FRONT_PROXY_CLIENT}" "client" "${subj}" "${CERT_DAYS}" "${FRONT_PROXY_CA}"
  log::info "${GREEN}updated ${BLUE}${FRONT_PROXY_CLIENT}.crt${NC}"

  # restart apiserver, controller-manager, scheduler and kubelet
  for item in "apiserver" "controller-manager" "scheduler"; do
    case $CRI in
      "docker")
        docker ps | awk '/k8s_kube-'${item}'/{print$1}' | xargs -r -I '{}' docker restart {} >/dev/null 2>&1 || true
        ;;
      "containerd")
        crictl ps | awk '/kube-'${item}'-/{print $(NF-1)}' | xargs -r -I '{}' crictl stopp {} >/dev/null 2>&1 || true
        ;;
    esac
    log::info "restarted ${item} with ${CRI}"
  done
  systemctl restart kubelet || true
  log::info "restarted kubelet"
}

main() {
  local node_type=$1

  # read the options
  ARGS=`getopt -o c: --long cri: -- "$@"`
  eval set -- "$ARGS"
  # extract options and their arguments into variables.
  while true
  do
    case "$1" in
      -c|--cri)
        case "$2" in
          "docker"|"containerd")
            CRI=$2
            shift 2
            ;;
          *)
            echo 'Unsupported cri. Valid options are "docker", "containerd".'
            exit 1
            ;;
        esac
        ;;
      --)
        shift
        break
        ;;
      *)
        echo "Invalid arguments."
        exit 1
        ;;
    esac
  done

  CERT_DAYS=3650

  KUBE_PATH=/etc/kubernetes
  PKI_PATH=${KUBE_PATH}/pki

  # master certificates path
  # apiserver
  CERT_CA=${PKI_PATH}/ca
  CERT_APISERVER=${PKI_PATH}/apiserver
  CERT_APISERVER_KUBELET_CLIENT=${PKI_PATH}/apiserver-kubelet-client
  CONF_CONTROLLER_MANAGER=${KUBE_PATH}/controller-manager
  CONF_SCHEDULER=${KUBE_PATH}/scheduler
  CONF_ADMIN=${KUBE_PATH}/admin
  CONF_SUPER_ADMIN=${KUBE_PATH}/super-admin
  CONF_KUBELET=${KUBE_PATH}/kubelet
  # front-proxy
  FRONT_PROXY_CA=${PKI_PATH}/front-proxy-ca
  FRONT_PROXY_CLIENT=${PKI_PATH}/front-proxy-client

  # etcd certificates path
  ETCD_CERT_CA=${PKI_PATH}/etcd/ca
  ETCD_CERT_SERVER=${PKI_PATH}/etcd/server
  ETCD_CERT_PEER=${PKI_PATH}/etcd/peer
  ETCD_CERT_HEALTHCHECK_CLIENT=${PKI_PATH}/etcd/healthcheck-client
  ETCD_CERT_APISERVER_ETCD_CLIENT=${PKI_PATH}/apiserver-etcd-client

  case ${node_type} in
  # etcd)
  # # update etcd certificates
  #   cert::update_etcd_cert
  # ;;
  master)
    # check certificates expiration
    cert::check_master_certs_expiration
    # backup $KUBE_PATH to $KUBE_PATH.old-$(date +%Y%m%d)
    cert::backup_file "${KUBE_PATH}"
    # update master certificates and kubeconf
    log::info "${GREEN}updating...${NC}"
    cert::update_master_cert
    log::info "${GREEN}done!!!${NC}"
    # check certificates expiration after certificates updated
    cert::check_master_certs_expiration
    ;;
  all)
    # check certificates expiration
    cert::check_all_expiration
    # backup $KUBE_PATH to $KUBE_PATH.old-$(date +%Y%m%d)
    cert::backup_file "${KUBE_PATH}"
    # update etcd certificates
    log::info "${GREEN}updating...${NC}"
    cert::update_etcd_cert
    # update master certificates and kubeconf
    cert::update_master_cert
    log::info "${GREEN}done!!!${NC}"
    # check certificates expiration after certificates updated
    cert::check_all_expiration
    ;;
  check)
    # check certificates expiration
    cert::check_all_expiration
    ;;
  *)
    log::err "unknown, unsupported cert type: ${node_type}, supported type: \"all\", \"master\""
    printf "Documentation: https://github.com/yuyicai/update-kube-cert
  example:
    '\033[32m./update-kubeadm-cert.sh all\033[0m' update all etcd certificates, master certificates and kubeconf
      /etc/kubernetes
      ├── admin.conf
      ├── super-admin.conf
      ├── controller-manager.conf
      ├── scheduler.conf
      ├── kubelet.conf
      └── pki
          ├── apiserver.crt
          ├── apiserver-etcd-client.crt
          ├── apiserver-kubelet-client.crt
          ├── front-proxy-client.crt
          └── etcd
              ├── healthcheck-client.crt
              ├── peer.crt
              └── server.crt

    '\033[32m./update-kubeadm-cert.sh master\033[0m' update only master certificates and kubeconf
      /etc/kubernetes
      ├── admin.conf
      ├── super-admin.conf
      ├── controller-manager.conf
      ├── scheduler.conf
      ├── kubelet.conf
      └── pki
          ├── apiserver.crt
          ├── apiserver-kubelet-client.crt
          └── front-proxy-client.crt
"
    exit 1
    ;;
  esac
}

main "$@"

image-20241017153706770

Master节点执行相应脚本
# 给update-kubeadm-cert.sh证书授权可执行权限
chmod +x update-kubeadm-cert.sh

image-20241017153757134

# 执行下面命令,修改证书过期时间,把时间延长到10年
./update-kubeadm-cert.sh all

image-20241017153900671

查询是否签发正常
# 在k8s-master1节点查询Pod是否正常,能查询出数据说明证书签发完成
kubectl get pods -n kube-system

image-20241017154042877

进一步确认是否延期成功
#  查看apiserver证书
[root@master01 ~]# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text  |grep Not
            Not Before: Oct 17 07:38:48 2024 GMT
            Not After : Oct 15 07:38:48 2034 GMT
您在 /var/spool/mail/root 中有新邮件
# 查看etcd证书
[root@master01 ~]# openssl x509 -in /etc/kubernetes/pki/apiserver-etcd-client.crt  -noout -text  |grep Not
            Not Before: Oct 17 07:38:48 2024 GMT
            Not After : Oct 15 07:38:48 2034 GMT
# 查看fron-proxy证书
[root@master01 ~]# openssl x509 -in /etc/kubernetes/pki/front-proxy-ca.crt  -noout -text  |grep Not
            Not Before: Sep 10 07:54:00 2024 GMT
            Not After : Sep  8 07:54:00 2034 GMT

image-20241017154247287

致谢

参考文章:

https://www.cnblogs.com/hujinzhong/p/14985449.html

https://blog.csdn.net/weixin_45248492/article/details/139200457?spm=1001.2014.3001.5506

点赞(0) 打赏

评论列表 共有 0 条评论

暂无评论

微信公众账号

微信扫一扫加关注

发表
评论
返回
顶部