zoukankan      html  css  js  c++  java
  • kubectl apply源码分析

    patch容易出现字段冲突

    近期在使用client-go对某个k8s原生资源进行patch操作时,出现了字段冲突导致的patch失败问题,具体是patch尝试修改资源的某个字段的类型,比如将readiness probe的类型从tcp修改为httpGet,patch时希望修改probe类型但被认为是一种追加动作,导致apiserver端验证错误不允许为一种类型的probe指定多个handler:

    当然,处理方式可以在patch数据中为要删除的readiness tcp probe加一个删除标记,这样patch请求到达apiserver的时候就可以被正确处理达到替换的目的:

    "spec": {
       "containers":[
          {
             "name":"xxx",
             "readinessProbe":{
                "exec":nil, // delete
                "httpGet":{ // add
                }
             }
          }
       }]
    }
    

    给我带来的疑惑是使用kubectl apply时为什么就没这个问题呢?

    kubectl apply使用3-way patch

    kubectl apply命令会在要apply的资源对象上添加last-apply-configuration,表示最近一次通过kubectl apply更新的资源清单,如果某个资源一直都是通过apply来更新,那么ast-apply-configuration与对象一致

    对于k8s原生的资源如deployment、pod等,kubectl apply时通过3-way patch生成strategicpatch类型的patch数据,其中:

    注意如果是crd资源,用的应该是jsonmergepatch.CreateThreeWayJSONMergePatch

    # staging/src/k8s.io/kubectl/pkg/cmd/apply/apply.go
    // 根据original、modified、current三方数据生成最终patch请求的数据
    if openapiPatch, err := strategicpatch.CreateThreeWayMergePatch(original, modified, current, lookupPatchMeta, p.Overwrite); err != nil {
    			fmt.Fprintf(errOut, "warning: error calculating patch from openapi spec: %v
    ", err)
    } else {
    			patchType = types.StrategicMergePatchType
    			patch = openapiPatch
    }
    

    current是集群中当前的资源数据:

        // info.Get通过RestClient请求api获取对象
    		if err := info.Get(); err != nil {
          // err是not found error,说明是首次创建
    			if !errors.IsNotFound(err) {
    				return cmdutil.AddSourceToErr(fmt.Sprintf("retrieving current configuration of:
    %s
    from server for:", info.String()), info.Source, err)
    			}
    			// Create the resource if it doesn't exist
    			// First, update the annotation used by kubectl apply
          // 如果集群中当前的对象没有last-apply-configuration注解,那么先用这个对象本身生存anno并更新到集群
    			if err := util.CreateApplyAnnotation(info.Object, unstructured.UnstructuredJSONScheme); err != nil {
    				return cmdutil.AddSourceToErr("creating", info.Source, err)
    			}
    		}
    

    modified是此次需要apply放入数据(比如-f指定的文件内容):

    		// Get the modified configuration of the object. Embed the result
    		// as an annotation in the modified configuration, so that it will appear
    		// in the patch sent to the server.
       // 可以看看这个方法具体的实现,会把自身encode之后放到自己的last-apply-configuration之中(覆盖可能已经存在的这个anno)
    		modified, err := util.GetModifiedConfiguration(info.Object, true, unstructured.UnstructuredJSONScheme)
    
    
    

    original是集群中当前资源的LastAppliedConfigAnnotation数据:

    // Retrieve the original configuration of the object from the annotation.
    original, err := util.GetOriginalConfiguration(obj)
    
    // GetOriginalConfiguration retrieves the original configuration of the object
    // from the annotation, or nil if no annotation was found.
    func GetOriginalConfiguration(obj runtime.Object) ([]byte, error) {
    	annots, err := metadataAccessor.Annotations(obj)
    	if err != nil {
    		return nil, err
    	}
    
    	if annots == nil {
    		return nil, nil
    	}
     // 直接取的annotation
    	original, ok := annots[v1.LastAppliedConfigAnnotation]
    	if !ok {
    		return nil, nil
    	}
    
    	return []byte(original), nil
    }
    

    有了这三方数据之后,strategicpatch.CreateThreeWayMergePatch方法就会产生最终要patch的数据

    • 根据集群中当前资源数据currentMap和此次要修改的数据Modified计算出那些字段是新增的,计算增量时忽略哪些要被删除的字段
      • 因为集群中的对象可能被修改过(人为或者某些组件)且这些修改不会更新last-apply-configuration anno,所以这里apply计算哪些字段是新增的时,就需要以集群当前状态和此次的apply数据modified来决定
    • 根据集群中当前资源的original(last-apply-configuration anno)数据和此次要修改的数据Modified计算出哪些字段是要删除的(设置为"-"),忽略增加的字段
      • kubectl apply认为冲突的字段应该通过相邻的两次apply操作来计算
    // CreateThreeWayMergePatch reconciles a modified configuration with an original configuration,
    // while preserving any changes or deletions made to the original configuration in the interim,
    // and not overridden by the current configuration. All three documents must be passed to the
    // method as json encoded content. It will return a strategic merge patch, or an error if any
    // of the documents is invalid, or if there are any preconditions that fail against the modified
    // configuration, or, if overwrite is false and there are conflicts between the modified and current
    // configurations. Conflicts are defined as keys changed differently from original to modified
    // than from original to current. In other words, a conflict occurs if modified changes any key
    // in a way that is different from how it is changed in current (e.g., deleting it, changing its
    // value). We also propagate values fields that do not exist in original but are explicitly
    // defined in modified.
    func CreateThreeWayMergePatch(original, modified, current []byte, schema LookupPatchMeta, overwrite bool, fns ...mergepatch.PreconditionFunc) ([]byte, error) {
      // 三方数据都反序列化为unstracture通用结构
    	originalMap := map[string]interface{}{}
    	if len(original) > 0 {
    		if err := json.Unmarshal(original, &originalMap); err != nil {
    			return nil, mergepatch.ErrBadJSONDoc
    		}
    	}
    
    	modifiedMap := map[string]interface{}{}
    	if len(modified) > 0 {
    		if err := json.Unmarshal(modified, &modifiedMap); err != nil {
    			return nil, mergepatch.ErrBadJSONDoc
    		}
    	}
    
    	currentMap := map[string]interface{}{}
    	if len(current) > 0 {
    		if err := json.Unmarshal(current, &currentMap); err != nil {
    			return nil, mergepatch.ErrBadJSONDoc
    		}
    	}
    
    	// The patch is the difference from current to modified without deletions, plus deletions
    	// from original to modified. To find it, we compute deletions, which are the deletions from
    	// original to modified, and delta, which is the difference from current to modified without
    	// deletions, and then apply delta to deletions as a patch, which should be strictly additive.
    	deltaMapDiffOptions := DiffOptions{
    		IgnoreDeletions: true,
    		SetElementOrder: true,
    	}
      // DiffOptions中IgnoreDeletions设置为true,根据集群中当前资源数据currentMap和此次要修改的数据计算出那些字段是新增的,
      // 计算增量时先忽略那些要被删除的
    	deltaMap, err := diffMaps(currentMap, modifiedMap, schema, deltaMapDiffOptions)
    	if err != nil {
    		return nil, err
    	}
    	deletionsMapDiffOptions := DiffOptions{
    		SetElementOrder:           true,
    		IgnoreChangesAndAdditions: true,
    	}
      // DiffOptions中IgnoreDeletions默认值为false,根据集群中当前资源的last-apply数据和此次要修改的数据计算出那些字段是要
      // 删除的,这里忽略增量的数据
      // 当有字段冲突时,这里会把original即上一次apply中的该字段标记为删除,deletionsMap中的值为nil
    	deletionsMap, err := diffMaps(originalMap, modifiedMap, schema, deletionsMapDiffOptions)
    	if err != nil {
    		return nil, err
    	}
    
    	mergeOptions := MergeOptions{}
      // 将deletionsMap和deltaMap做一次合并,生成最终要patch的数据
    	patchMap, err := mergeMap(deletionsMap, deltaMap, schema, mergeOptions)
    	if err != nil {
    		return nil, err
    	}
    
    	return json.Marshal(patchMap)
    }
    
    func diffMaps(original, modified map[string]interface{}, schema LookupPatchMeta, diffOptions DiffOptions) (map[string]interface{}, error) {
      // 记录结果
    	patch := map[string]interface{}{}
    
    	// Compare each value in the modified map against the value in the original map
      // 遍历originalMap这个unstrctureMap的每一个key
    	for key, modifiedValue := range modified {
    		originalValue, ok := original[key]
    		if !ok {
    			// Key was added, so add to patch
          // 如果value不存在于originalMap,但是存在于modifiedMap,并且IgnoreChangesAndAdditions为false
    			if !diffOptions.IgnoreChangesAndAdditions {
            // 结果添加modifiedMap中的这个kv
    				patch[key] = modifiedValue
    			}
    			continue
    		}
    		
        // original和modified中都有value,就看value是不是同一种类型
    		if reflect.TypeOf(originalValue) != reflect.TypeOf(modifiedValue) {
    			// Types have changed, so add to patch
          // 类型一样并且IgnoreChangesAndAdditions为false,那么结果添加modifiedMap中的这个kv
    			if !diffOptions.IgnoreChangesAndAdditions {
    				patch[key] = modifiedValue
    			}
    			continue
    		}
    
    		// Types are the same, so compare values
        // original和modified中都有value,就看value是同一种类型
        // 那么根据具体的类型,调用handleMapDiff或handleSliceDiff处理
    		switch originalValueTyped := originalValue.(type) {
        // value的类型是一个复合结构
    		case map[string]interface{}:
    			modifiedValueTyped := modifiedValue.(map[string]interface{})
    			err = handleMapDiff(key, originalValueTyped, modifiedValueTyped, patch, schema, diffOptions)
        // value的类型是一个slice切片结构
    		case []interface{}:
    			modifiedValueTyped := modifiedValue.([]interface{})
    			err = handleSliceDiff(key, originalValueTyped, modifiedValueTyped, patch, schema, diffOptions)
    		default:
        // 既不是map也不是slice,那么直接用modifiedValue替换originalValue
    			replacePatchFieldIfNotEqual(key, originalValue, modifiedValue, patch, diffOptions)
    		}
    		if err != nil {
    			return nil, err
    		}
    	}
    	// 如果ignoreDeletions为false,那么遍历originalMap的每一个key,如果modefiedMap中不存在value,那么在最终的结果中
      // 标记该key为需要删除
    	updatePatchIfMissing(original, modified, patch, diffOptions)
    	return patch, nil
    }
    

    从上面的分析可以看出,kubect在apply时通过3-way patch的方式,可以计算出哪些字段是要新增的,哪些字段是要被删除的,以避免冲突的出现,如果original中的数据(last-apply)与modifed不能正确计算出要被删除的字段,也会出现apply失败的问题,比如资源通过kubectl create创建则没有last-apply-configuration注解,这个时候如果修改字段的值类型,即使通过kubectl apply也会失败。

  • 相关阅读:
    【codeforces 709D】Recover the String
    【codeforces 707C】Pythagorean Triples
    【hdu 1043】Eight
    Android提高第十二篇之蓝牙传感应用
    Method of offloading iSCSI TCP/IP processing from a host processing unit, and related iSCSI TCP/IP offload engine
    VS2015 IIS Express 无法启动 解决办法
    【转】译文:.net 程序员易犯的7个通病
    【转】译文:.net 程序员易犯的7个通病
    【转】译文:.net 程序员易犯的7个通病
    几个故事告诉你,火热的区块链究竟是什么?
  • 原文地址:https://www.cnblogs.com/orchidzjl/p/15086579.html
Copyright © 2011-2022 走看看