基于karmada的多集群管理（二）
Kubernetes
k8s karmada
发布日期:   2023-04-10
文章作者:   XieJiayi
文章字数:   1.8k
阅读时长:   8 分
阅读次数:  

基于Karmada的多集群管理-调度(二)

介绍

上文主要介绍了karmada集群的快速搭建以及基本的用例的下发demo,本文主要详细介绍karmada集群中调度传播策略propagationpolicies、差异性配置overridepolicies的资源介绍。

Propagationpolicy、ClusterPropagationpolicy(分发策略)

部署应用到指定集群

.spec.placement.clusterAffinity 字段可以配置分发策略规则分发到指定的集群中，若没有配置则所有集群都可以作为候选调度。

主要有4中配置方式

  • LabelSelector（标签选择）
  • FieldSelector
  • ClusterNames
  • ExcludeClusters

1. LabelSelector

LabelSelector使用标签过滤选择成员集群，kubectl get cluster cluster121 -oyaml 中*metav1.LabelSelector

示例如下：

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: test-propagation
spec:
  #...
  placement:
    clusterAffinity:
      labelSelector:
        matchLabels:
          location: us
    #...

也可以配置这种：

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: test-propagation
spec:
  #...
  placement:
    clusterAffinity:
      labelSelector:
        matchExpressions:
        - key: location
          operator: In
          values:
          - us
    #...

operator配置规则参考k8s:https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements

2. FieldSelector

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
spec:
  #...
  placement:
    clusterAffinity:
      fieldSelector:
        matchExpressions:
        - key: provider
          operator: In
          values:
          - huaweicloud
        - key: region
          operator: NotIn
          values:
          - cn-south-1
    #...

3.ClusterNames

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
spec:
  #...
  placement:
    clusterAffinity:
      clusterNames:
        - member1
        - member2
    #...

4.ExcludeClusters

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
spec:
  #...
  placement:
    clusterAffinity:
      exclude:
        - member1
        - member3
    #...

基于污点和容忍的调度方式

.spec.placementPropagationPolicy容忍字段。像kubernetes一样，容忍需要与集群上的污点一起使用。在集群上设置一个或多个污点后，工作负载就不能在这些集群上调度或运行，除非策略明确声明这些污点是可以容忍的。业力目前支持影响为NoSchedule和NoExecute的污染。

手动给集群打污点

karmadactl taint clusters foo dedicated=special-user:NoSchedule

设置容忍策略

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
spec:
  #...
  placement:
    clusterTolerations:
    - key: dedicated
      value: special-user
      Effect: NoSchedule

副本调度的多种策略

.spec.placement.plicaScheduling分发具有副本的资源时处理副本数量的调度策略

  • Duplicated策略
  • Divided 策略

Aggregated 策略

Aggregated 策略表示将副本调度到尽可能少的目标集群上，例如目标集群有三个，但是第一个目标集群拥有足够的资源调度到所有的副本，则副本会全部调度到第一个目标集群上，如图所示：

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
spec:
  #...
  placement:
    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Duplicated

Weighted 策略

Weighted 策略具体设置通过 .spec.placement.weightPreference 字段设置，如果 weightPreference 不设置，则默认给所有目标集群加相同的静态权重 1。支持staticWeightList和dynamicWeight 静态权重按照比例分配，动态权重按照可用副本分配

# 按照权重分配到不用的成员集群
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
spec:
  #...
  placement:
    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Divided
      weightPreference:
        staticWeightList:
          - targetCluster:
              clusterNames:
                - member1
            weight: 1
          - targetCluster:
              clusterNames:
                - member2
            weight: 1
# 按照可用副本数进行分配
#例如，调度程序选择了3个集群（A/B/C），应将12个副本分配给它们。根据集群资源建模，我们得到A、B、C的最大可用副本分别为6、12、18。因此，集群A:B:C的权重将为6:12:18（等于1:2:3）。最后，分配将是“A：2，B：4，C：6”。
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
spec:
  #...
  placement:
    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Divided
      weightPreference:
        dynamicWeight: AvailableReplicas

配置传播策略的优先级

配置显性优先级

spec.priority配置传播策略PropagationPolicy 的显性优先级，更大值代表更高的优先级

Note: 未指定默认为0

示例：

# highexplicitpriority.yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: propagation-high-explicit-priority
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      labelSelector:
        matchLabels:
          app: nginx
  priority: 2
  placement:
    clusterAffinity:
      clusterNames:
        - member1
---
# lowexplicitpriority.yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: propagation-low-explicit-priority
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      labelSelector:
        matchLabels:
          app: nginx
  priority: 1
  placement:
    clusterAffinity:
      clusterNames:
        - member2
---
# defaultexplicitpriority.yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: propagation-low-explicit-priority
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      labelSelector:
        matchLabels:
          app: nginx
  placement:
    clusterAffinity:
      clusterNames:
        - member3

最终负责则会调度到member1

配置隐性优先级

spec.resourceSelectors配置传播策略PropagationPolicy 的隐性优先级，以下优先级从下到大为：

  • 传播策略资源的name和标签选择器为空
  • 存在传播策略资源标签选择器
  • 存在传播策略资源名称
# emptymatch.yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: propagation-emptymatch
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
  placement:
    clusterAffinity:
      clusterNames:
        - member1
---
# labelselectormatch.yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: propagation-labelselectormatch
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      labelSelector:
        matchLabels:
          app: nginx
  placement:
    clusterAffinity:
      clusterNames:
        - member2
---
# namematch.yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: propagation-namematch
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx
  placement:
    clusterAffinity:
      clusterNames:
        - member3

最终会调度到member3

存在相同优先级

根据 metadata.name的字母顺序来决定。如xxx-a-xxx 优先于 xxx-b-xxx.

# higher-alphabetical.yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: propagation-b-higher-alphabetical
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx
  priority: 3
  placement:
    clusterAffinity:
      clusterNames:
        - member1
---
# lower-alphabetical.yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: propagation-a-lower-alphabetical
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx
  priority: 3
  placement:
    clusterAffinity:
      clusterNames:
        - member2

OverridePolicy、ClusterOverridePolicy（差异性配置）

差异性配置主要为两部分配置：resourceSelectors集群选择、overrideRules覆盖规则

目标集群

主要与Propagationpolicy类似，包含4中选择器

  • labelSelector: 基于标签方式
  • fieldSelector: 基于字段方式；目前只支持cluster.spec.provider,cluster.spec.zone,cluster.spec.region三种
  • clusterNames: 基于名称
  • exclude: 基于忽略集群

labelSelector

apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: example
spec:
  #...
  overrideRules:
    - targetCluster:
        labelSelector:
          matchLabels:
            cluster: member1 
      overriders:
      #...

fieldSelector

apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: example
spec:
  #...
  overrideRules:
    - targetCluster:
        fieldSelector:
          matchExpressions:
            - key: region
              operator: In
              values:
                - cn-north-1
      overriders:
      #...
----

apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: example
spec:
  #...
  overrideRules:
    - targetCluster:
        fieldSelector:
          matchExpressions:
            - key: provider
              operator: In
              values:
                - aws
      overriders:
      #...
      
---
apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: example
spec:
  #...
  overrideRules:
    - targetCluster:
        fieldSelector:
          matchExpressions:
            - key: zone
              operator: In
              values:
                - us
      overriders:
      #...

clusterNames

apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: example
spec:
  #...
  overrideRules:
    - targetCluster:
        clusterNames:
          - member1
      overriders:
      #...

exclude

apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: example
spec:
  #...
  overrideRules:
    - targetCluster:
        exclude:
          - member1
      overriders:
      #...

重写规则

Karmada支持一下几种方式的重写：

  • ImageOverrider: 覆盖工作负载的镜像
  • CommandOverrider: 覆盖工作负载的命令
  • ArgsOverrider: 覆盖工作负载的参数
  • LabelsOverrider: 覆盖工作负载的标签
  • AnnotationsOverrider: 覆盖工作负载的注解
  • PlaintextOverrider: 覆盖任何资源通用方法

ImageOverrider

component标签[registry/]repository[:tag|@digest]涉及为：

  • Registry
  • Repository
  • Tagest

操作方式为：

  • add
  • remove
  • replace

Examples

创建一个样例：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  #...
spec:
  template:
    spec:
      containers:
        - image: myapp:1.0.0
          name: myapp

Example 1

apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: example
spec:
  #...
  overrideRules:
    - overriders:
        imageOverrider:
          - component: Registry
            operator: add
            value: test-repo
#add Registry 
      containers:
        - image: test-repo/myapp:1.0.0
          name: myapp

Example 2

apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: example
spec:
  #...
  overrideRules:
    - overriders:
        imageOverrider:
          - component: Repository
            operator: replace
            value: myapp2
# replace repository
      containers:
        - image: myapp2:1.0.0
          name: myapp

Example 3

apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: example
spec:
  #...
  overrideRules:
    - overriders:
        imageOverrider:
          - component: Tag
            operator: remove
# remove tag
      containers:
        - image: myapp
          name: myapp

CommandOverrider

覆盖/spec/template/spec/containers/0/command命令

  • add
  • remove
apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: example
spec:
  #...
  overrideRules:
    - overriders:
        commandOverrider:
          - containerName: myapp
            operator: add
            value:
              - --cluster=member1
              
---

apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: example
spec:
  #...
  overrideRules:
    - overriders:
        commandOverrider:
          - containerName: myapp
            operator: remove
            value:
              - --parameter1=foo

ArgsOverrider

覆盖参数/spec/template/spec/containers/0/args ，使用方式类似于CommandOverrider

  • add
  • remove

LabelsOverrider

覆盖标签

  • add
  • remove
  • replace
apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: example
spec:
  #...
  overrideRules:
    - overriders:
        labelsOverrider:
          - operator: add
            value: 
              bar: bar  # It will be added to labels
          - operator: replace
            value: 
              foo: exist # "foo: foo" will be replaced by "foo: exist"
          - operator: remove
            value: 
              baz: baz   # It will be removed from labels

AnnotationsOverrider

注解标签

  • add
  • remove
  • replace

PlaintextOverrider

通用资源处理器

  • add
  • remove
  • replace
#Example 
apiVersion: v1
kind: ConfigMap
metadata:
  name: myconfigmap
  #...
data:
  example: 1
  
---
#Overrider
apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: example
spec:
  #...
  overrideRules:
    - overriders:
        plaintext:
          - path: /data/example
            operator: replace
            value: 2
            
---
#Result
apiVersion: v1
kind: ConfigMap
metadata:
  name: myconfigmap
  #...
data:
  example: 2

   转载规则


《基于karmada的多集群管理（二）》 by XieJiayi is licensed under a 知识共享署名 4.0 国际许可协议 许可协议。转载请注明来源
 上一篇
Metallb结合Ingress的使用 Metallb结合Ingress的使用
Metallb结合Ingress的使用一、Metallb1.简介 Kubernetes不提供网络负载均衡器的实现（LoadBalancer类型的服务）用于裸机集群。Kubernetes附带的Network LB的实现都是调用各种IaaS平台
2023-05-31 k8s
Kubernetes
下一篇 
基于Karmada的多集群管理（一） 基于Karmada的多集群管理（一）
基于Karmada的多集群管理(一)简介Karmada（Kubernetes Armada）是基于Kubernetes原生API的多集群管理系统。在多云和混合云场景下，Karmada提供可插拔，全自动化管理多集群应用，实现多云集中管理、高可
2023-04-07 k8s karmada
Kubernetes
  目录