使用Jenkins和單個(gè)模板部署多個(gè)Kubernetes組件

前言

在持續(xù)集成和部署中，我們通常需要部署多個(gè)實(shí)例或組件到Kubernetes集群中。通過Jenkins的管道腳本，我們可以自動(dòng)化這個(gè)過程。在本文中，我將演示如何使用Jenkins Pipeline及單個(gè)YAML模板文件（.tpl）來部署多個(gè)類似的Kubernetes組件，而不需要為每個(gè)組件提供單獨(dú)的模板文件。

問題背景

Jenkins Pipeline 腳本優(yōu)化實(shí)踐：從繁瑣到簡(jiǎn)潔批量生成 Kubernetes 部署模板：從 1 到20順序模板

pipeline {
    agent none // Use none at the top level, each stage will define its own agent.


    environment {
        REGISTRY = "xxxx/master-metaspace"
        KUBE_CONFIG = "--namespace=master-metaspace --context=master"
        KUBE_YAML_PATH = "/home/jenkins/workspace/yaml/master-metaspace"
        // Assume that 'data' is defined elsewhere or injected as a parameter.
        BASE_WORKSPACE = "xxxxxxx" // 定義一個(gè)基礎(chǔ)工作空間路徑
    }


    stages {
        stage("GetCode") {
            agent { label "build01" }
            steps {
                script {
                    checkout scm: [
                        $class: 'GitSCM',
                        branches: [[name: env.branchName]],
                        extensions: [[$class: 'CloneOption', depth: 1, noTags: false, shallow: true]],
                        userRemoteConfigs: [[credentialsId: 'xxxx', url: env.gitHttpURL]]
                    ]
                }
            }
        }
        
        stage("Docker Builds") {
            parallel {
                stage('Build dataloader-game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'dataloader', value: 'true' }
                    steps {
                        buildAndPushDockerImage("dataloader-game-ucenter", env.data, env.BASE_WORKSPACE)
                    }
                }
                stage('Build datawriter-game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'datawriter', value: 'true' }
                    steps {
                        buildAndPushDockerImage("datawriter-game-ucenter", env.data, env.BASE_WORKSPACE)
                    }
                }
                stage('Build game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'game-ucenter', value: 'true' }
                    steps {
                        buildAndPushDockerImage("game-ucenter", env.data, env.BASE_WORKSPACE)
                    }
                }
            }
        }
        
        stage('Development Deployment') {
            parallel {
                stage("Deploy datawriter-game-ucenter") {
                    when { environment name: 'datawriter-game-ucenter', value: 'true' }
                    agent { label  "huaweiyun-xx" }
                    steps {
                        deployToKubernetes("datawriter-game-ucenter")
                    }
                }
                stage("Deploy dataloader-game-ucenter") {
                    when { environment name: 'dataloader', value: 'true' }
                    agent { label  "huaweiyun-xx" }
                    steps {
                        deployToKubernetes("dataloader-game-ucenter")
                    }
                }
                stage("Deploy game-ucenter") {
                    when { environment name: 'game-ucenter', value: 'true' }
                    agent { label  "huaweiyun-xx" }
                    steps {
                        deployToKubernetes("game-ucenter-1")
                        deployToKubernetes("game-ucenter-2")
                        deployToKubernetes("game-ucenter-3")
                        deployToKubernetes("game-ucenter-4")
                        ............................
                    }
                }
            }
        }
    }
}


// Define methods outside pipeline to avoid repetition


def buildAndPushDockerImage(String imageName, String tag, String workspacePath) {
    sh "cd ${workspacePath} && echo 'Current directory: $(pwd)'" // 使用基礎(chǔ)工作空間變量
    sh "cd ${workspacePath}/${imageName}&& docker build --build-arg NODE_ENV=$imageName -t $REGISTRY/$imageName:$tag ."
    withCredentials([usernamePassword(credentialsId: 'xxxxx', passwordVariable: 'dockerPassword', usernameVariable: 'dockerUser')]) {
        sh "docker login -u $dockerUser -p $dockerPassword $REGISTRY"
        sh "docker push $REGISTRY/$imageName:$tag"
    }
}


def deployToKubernetes(String kubernetesComponent) {
    String templateFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.tpl"
    String outputFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.yaml"
    sh "sed -e 's/{data}/$data/g' $templateFile > $outputFile"
    sh "sudo kubectl apply -f $outputFile $KUBE_CONFIG"
}

默認(rèn)jenkins pipeline如上，我們有多個(gè)相似的游戲用戶中心服務(wù)game-ucenter-*運(yùn)行在Kubernetes集群中，它們都使用非常相似的Kubernetes YAML配置文件，配置文件之間的差異主要是一些標(biāo)識(shí)符的不同（例如，服務(wù)的序號(hào)）。在傳統(tǒng)的做法中，維護(hù)一系列幾乎一樣的模板文件（如game-ucenter-1.tpl, game-ucenter-2.tpl 等）將非常低效且易出錯(cuò)。

為了精簡(jiǎn)流程和提高效率，我們需要一個(gè)方法來通過單一模板生成多個(gè)配置文件，并由此部署多個(gè)不同的服務(wù)實(shí)例。

解決方案

使用Jenkins Pipeline中的sed命令和循環(huán)結(jié)構(gòu)，我們可以從單一模板生成多個(gè)Kubernetes配置文件，并相應(yīng)地部署每個(gè)服務(wù)實(shí)例。參照generate_templates.sh腳本

#!/bin/bash


# Define the name of the template file.
TEMPLATE_FILE="game-ucenter.tpl"


# Check if the template file exists.
if [ ! -f "$TEMPLATE_FILE" ]; then
    echo "Template file $TEMPLATE_FILE does not exist."
    exit 1
fi


# Loop to create files from game-ucenter-2 to pvp-game-20 based on the template.
for i in $(seq 1 20); do
    # Define the name of the new file.
    NEW_FILE="game-ucenter-${i}.yaml"
    
    # Copy the template to the new file.
    cp $TEMPLATE_FILE $NEW_FILE
    
    # Use 'sed' to replace 'game-ucenter-1' with 'game-ucenter-N' and save inline (-i option).
    sed -i "s/game-ucenter/game-ucenter-${i}/g" $NEW_FILE




    echo "Created file: $NEW_FILE"
done


echo "All files created successfully."

步驟 1: 定義Jenkins Pipeline

在我們的Jenkins腳本中，我們首先定義了基礎(chǔ)環(huán)境變量和兩個(gè)函數(shù)：buildAndPushDockerImage 和 deployToKubernetes。這些函數(shù)將用于構(gòu)建Docker鏡像并部署到Kubernetes

def buildAndPushDockerImage(String imageName, String tag, String workspacePath) {
    sh "cd ${workspacePath} && echo 'Current directory: $(pwd)'" // 使用基礎(chǔ)工作空間變量
    sh "cd ${workspacePath}/${imageName}&& docker build --build-arg NODE_ENV=$imageName -t $REGISTRY/$imageName:$tag ."
    withCredentials([usernamePassword(credentialsId: 'xxx', passwordVariable: 'dockerPassword', usernameVariable: 'dockerUser')]) {
        sh "docker login -u $dockerUser -p $dockerPassword $REGISTRY"
        sh "docker push $REGISTRY/$imageName:$tag"
    }
}


def deployToKubernetes(String kubernetesComponent) {
    String templateFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.tpl"
    String outputFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.yaml"
    sh "sed -e 's/{data}/$data/g' $templateFile > $outputFile"
    sh "sudo kubectl apply -f $outputFile $KUBE_CONFIG"
}

步驟 2: 修改deployToKubernetes函數(shù)

接下來，我們需要修改deployToKubernetes函數(shù)，以便它能夠接受組件名稱，并使用單一模板文件創(chuàng)建具體的配置文件。

def deployToKubernetes(String kubernetesComponent, int instance=1, int totalInstances=1) {
    // 檢查實(shí)例值
    if (instance < 1) {
        error("實(shí)例數(shù)必須大于0")
    }


    // 根據(jù) instance 的值來定義資源的名稱和文件名
    String nameSuffix = totalInstances > 1 ? "-${instance}" : "" // 總是添加后綴，除非只有一個(gè)實(shí)例
    String outputFileName = "${kubernetesComponent}${nameSuffix}.yaml"
    String templateFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.tpl"
    String outputFile = "${KUBE_YAML_PATH}/${outputFileName}"
    String nameReplacement = "${kubernetesComponent}${nameSuffix}"


    sh """
        cat "${templateFile}" 
        | sed 's/{data}/${data}/g' 
        | sed 's/name: ${kubernetesComponent}/name: ${nameReplacement}/g' 
        | sed 's/app: ${kubernetesComponent}/app: ${nameReplacement}/g' 
        > "${outputFile}"
    """


    // 使用 KUBE_CONFIG 應(yīng)用 Kubernetes 配置
    sh "kubectl apply -f ${outputFile} ${KUBE_CONFIG}"
}

對(duì)于單實(shí)例的業(yè)務(wù)，例如Deploy dataloader-game-ucenter，我們不需要傳遞實(shí)例編號(hào)。

stage("Deploy dataloader-game-ucenter") {
    when { environment name: 'dataloader', value: 'true' }
    agent { label  "huaweiyun-xx" }
    steps {
        deployToKubernetes("dataloader-game-ucenter")
    }
}

對(duì)于多實(shí)例。我這里生成 規(guī)則優(yōu)點(diǎn)強(qiáng)迫癥了。如果多實(shí)例我生成的規(guī)則要求符合game-ucenter-1,game-ucenter-2,game-ucenter-3......順序，當(dāng)單個(gè)實(shí)例的時(shí)候則保持原來的不加標(biāo)簽：

                stage("Deploy game-ucenter") {
                    when { environment name: 'game-ucenter', value: 'true' }
                    agent { label  "k8s-node-06" }
                    steps {
                        script {
                          int instances = 2 // 假設(shè)我們有2個(gè)實(shí)例
                          for (int i = 1; i <= instances; i++) {
                            def componentName = "game-ucenter"
                            deployToKubernetes("game-ucenter", i, instances)


                    }
                }
            }
        }

步驟 3: 準(zhǔn)備模板文件

我們的模板文件game-ucenter.tpl將包含通用的Kubernetes服務(wù)或部署定義，使用占位符game-ucenter-1game-ucenter-2來標(biāo)識(shí)應(yīng)該被替換的地方。

# game-ucenter-1.tpl (示例部分)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: game-ucenter
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: game-ucenter
  template:
    metadata:
      labels:
        app: game-ucenter
    spec:
      containers:
        - name: game-ucenter
          image: xxxx/xxx/game-ucenter:{data}
          envFrom:
          - configMapRef:
              name: deploy
          ports:
            - containerPort: 80
          resources:
            requests:
              memory: "4096M"
              cpu: "2000m"
            limits:
              memory: "4096M"
              cpu: "2000m" 
          livenessProbe:
            httpGet:
              scheme: HTTP
              path: /test.html
              port: 80
            initialDelaySeconds: 20
            periodSeconds: 120
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              scheme: HTTP
              path: /test.html
              port: 80
            initialDelaySeconds: 20
            periodSeconds: 120
      imagePullSecrets:                                              
        - name: xxx
---


apiVersion: v1
kind: Service
metadata:
  name: game-ucenter
  labels:
    app: game-ucenter
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: game-ucenter


# ...

步驟 4: 執(zhí)行Jenkins Pipeline

當(dāng)Jenkins Pipeline運(yùn)行到"Development Deployment"階段時(shí)，它將循環(huán)創(chuàng)建和應(yīng)用game-ucenter-1.yaml到game-ucenter-2.yaml的配置文件，從而部署2個(gè)game-ucenterdeployment服務(wù)實(shí)例。

并保證單個(gè)實(shí)例的原有命名規(guī)則：

通過這一方法，我們不再需要為每個(gè)服務(wù)實(shí)例維護(hù)一個(gè)單獨(dú)的模板文件，而是可以通過一個(gè)模板文件和Jenkins Pipeline的自動(dòng)化來簡(jiǎn)化服務(wù)部署工作。這樣做不僅提升了效率，也降低了出錯(cuò)的風(fēng)險(xiǎn)。

注意：

以上代碼和命令為示例性質(zhì)，可能需要根據(jù)您具體的Jenkins環(huán)境和Kubernetes集群進(jìn)行相應(yīng)的調(diào)整。在生產(chǎn)環(huán)境中部署之前，請(qǐng)確保進(jìn)行充分的測(cè)試。

審核編輯：湯梓紅