kubernets controller 和 CRD 具体组件分析

(dlv) b k8s.io/sample-controller/pkg/client/informers/externalversions.(*sharedInformerFactory).InformerFor
(dlv) b  k8s.io/sample-controller/vendor/k8s.io/client-go/informers.(*sharedInformerFactory).InformerFor
(dlv) b k8s.io/sample-controller/vendor/k8s.io/client-go/informers.(*sharedInformerFactory).Start
(dlv) b k8s.io/sample-controller/pkg/client/informers/externalversions.(*sharedInformerFactory).Start

(dlv) bp
Breakpoint unrecovered-panic at 0x42c850 for runtime.fatalpanic() /usr/local/go/src/runtime/panic.go:681 (0)
        print runtime.curg._panic.arg
Breakpoint 1 at 0xf432b8 for k8s.io/sample-controller/pkg/client/informers/externalversions.(*sharedInformerFactory).InformerFor() ./pkg/client/informers/externalversions/factory.go:147 (2)
Breakpoint 2 at 0xefbc38 for k8s.io/sample-controller/vendor/k8s.io/client-go/informers.(*sharedInformerFactory).InformerFor() ./vendor/k8s.io/client-go/informers/factory.go:162 (2)
Breakpoint 3 at 0xefb888 for k8s.io/sample-controller/vendor/k8s.io/client-go/informers.(*sharedInformerFactory).Start() ./vendor/k8s.io/client-go/informers/factory.go:126 (0)
Breakpoint 4 at 0xf42f08 for k8s.io/sample-controller/pkg/client/informers/externalversions.(*sharedInformerFactory).Start() ./pkg/client/informers/externalversions/factory.go:111 (0)

编写控制器
Kubernetes控制器是一个主动调和过程。也就是说，它会为观察期望的物体状态，它也会观察实际的物体状态。然后，它发送指令以尝试使当前状态符合所需的状态。

最简单的实现是循环：

for {
  desired：= getDesiredState（）
  current：= getCurrentState（）
  makeChanges（desired：current）
}
watches等都只是对这种逻辑的优化。

指导：
当您编写控制器时，很少有指导原则可以帮助您确保获得所需的结果和性能。

1. 一次操作一个item。
如果你使用workqueue.Interface，你将能够为特定资源的更改排队，然后在多个“worker”的gofuncs中弹出它们，并保证两个gofunc不会在同一时刻在同一个item上运行。

许多controller必须触发多个资源（我需要“检查X是否Y更改”），但几乎所有控制器都可以根据关系将这些资源折叠成“检查此X”的队列。例如，ReplicaSet控制器需要对正在删除的pod做出反应，但它通过查找相关的ReplicaSet并对其进行排队来实现。

2. 资源之间的随机排序。
当controller排队多种类型的资源时，无法保证在这些资源之间进行排序。
Distinct的watches更新是独立的。
即使先“创建资源A / X”，在“创建资源B / Y”的顺序，controller可能观察到“创建资源B / Y”和“创建资源A / X”。

3. level驱动，而不是edge驱动。controller可能会关闭一段不确定的时间，然后再次运行。
如果API对象出现时标记为true，那么您不能指望看到它从false变为true，您只能看到它是true的。即使是API watch也会遇到这个问题，所以请确保您不要指望看到变化，除非您的控制器还标记了它最后在对象状态中做出决定的信息。

4. 使用SharedInformers。 SharedInformers提供钩子以接收特定资源的添加，更新和删除通知。它还提供便利的功能，用于访问共享缓存并确定何时启动缓存。

使用https://git.k8s.io/kubernetes/staging/src/k8s.io/client-go/informers/factory.go中的工厂方法确保您与其他人共享同一个缓存实例。

这节省了我们与API服务器的连接，服务器端的重复序列化成本，controller端的重复反序列化成本以及controller端的重复高速缓存成本。

您可能会看到其他机制，如反射器和deltafifos驱动controller。那些是我们后来用于构建SharedInformer的旧机制。您应该避免在新控制器中使用它们。

5. 永远不要改变原始对象！缓存在控制器之间共享，这意味着如果你改变对象的“副本”（实际上是引用或浅拷贝），你将搞乱其他controller（而不仅仅是你自己的controller）。

最常见的失败点是制作浅层副本，然后改变map，例如Annotations。使用api.Scheme.Copy进行深层复制。

6. 等待二级缓存。许多controller具有主要和次要资源。主要资源是您要更新状态的资源。辅助资源是您将管理（创建/删除）或用于查找的资源。

在启动主同步功能之前，使用framework.WaitForCacheSync函数等待辅助缓存。这将确保像ReplicaSet的Pod计数不处理导致颠簸的已知过时信息。

7. 系统中还有其他参与者。你没有改变一个对象并不意味着其他人没有。

不要忘记当前状态可能随时改变 - 仅仅观察所需的状态是不够的。如果您使用处于期望状态的对象的缺失来指示应删除处在当前状态的内容，请确保您的observation 代码中没有错误（例如，在缓存填充之前执行操作）。

8. 将错误渗透到顶层以实现一致的重新排队。我们有一个workqueue.RateLimitingInterface，允许通过合理的backoffs进行简单的重新排队。

当需要重新排队时，主controller func应该返回错误。如果不是，则应使用utilruntime.HandleError并返回nil。这使得reviewers可以非常轻松地检查错误处理案例，并确信您的controller不会意外丢失它应该应重试的东西。

9. watch和informers(线人？)将“同步”。他们会定期将群集中的每个匹配对象传递给Update方法。这对于可能需要在对象上采取其他额外操作的情况很有用，但有时您知道不会有更多工作要做。

如果您确定在没有新的变化，不需要重新排队列表时，则可以比较新旧对象的resource版本。如果它们相同，则跳过requeuing。这样做时要小心。如果您跳过故障时跳过了重新requeue item，则可能会失败，不是requeue，然后再也不会重试该item。

10. 如果controller正在协调的主要资源的status中支持ObservedGeneration，只要改值不匹配，请确保正确设置为metadata.Generation。

这使客户端知道controller已处理资源。确保您的控制器是负责该资源的主controller，否则如果您需要通过自己controller进行交流观察，则需要在资源的状态中创建不同类型的ObservedGeneration。

11. 考虑使用资源owner references，资源所有者引用可导致创建其他资源（例如，ReplicaSet导致创建Pod）。因此，一旦删除了控制器管理的资源，就可以确保对子资源进行垃圾回收。有关owner references的更多信息，请阅读此处（https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/controller-ref.md）。

注意您采用的方式。当父或子资源被标记为删除时，不要使用子资源。如果您正在为资源使用缓存，则可能需要使用直接API读取来绕过它，以防您发现已为其中一个子更新了owner reference。因此，您确保您的collector不与garbage collector竞争。

有关更多信息，请参阅k8s.io/kubernetes/pull/42938。

1. Reflector

debug:

(dlv) b main.main
Breakpoint 1 set at 0xf471eb for main.main() ./main.go:40
(dlv) b Reflector.ListAndWatch
Breakpoint 2 set at 0xe74fcb for k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch() ./vendor/k8s.io/client-go/tools/cache/reflector.go:168

图中 2) Add Objects 

主要实现ListAndWatch （通过 channel 和 receive通信）

// Run starts a watch and handles watch events. Will restart the watch if it is closed.
// Run will exit when stopCh is closed.
func (r *Reflector) Run(stopCh <-chan struct{}) {
	klog.V(3).Infof("Starting reflector %v (%s) from %s", r.expectedType, r.resyncPeriod, r.name)
	wait.Until(func() {
		if err := r.ListAndWatch(stopCh); err != nil {
			utilruntime.HandleError(err)
		}
	}, r.period, stopCh)
}

先list

要解决URL:

list
k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/runtime.Object(*k8s.io/sample-controller/pkg/apis/samplecontroller/v1alpha1.FooList) *{
        TypeMeta: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.TypeMeta {Kind: "", APIVersion: ""},
        ListMeta: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.ListMeta {
                SelfLink: "/apis/samplecontroller.k8s.io/v1alpha1/foos",
                ResourceVersion: "204818",
                Continue: "",},
        Items: []k8s.io/sample-controller/pkg/apis/samplecontroller/v1alpha1.Foo len: 1, cap: 1, [
                (*k8s.io/sample-controller/pkg/apis/samplecontroller/v1alpha1.Foo)(0xc000116280),
        ],}

list items

items
[]k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/runtime.Object len: 1, cap: 1, [
        *k8s.io/sample-controller/pkg/apis/samplecontroller/v1alpha1.Foo {
                TypeMeta: (*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.TypeMeta)(0xc000116280),
                ObjectMeta: (*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.ObjectMeta)(0xc0001162a0),
                Spec: (*k8s.io/sample-controller/pkg/apis/samplecontroller/v1alpha1.FooSpec)(0xc000116388),
                Status: (*k8s.io/sample-controller/pkg/apis/samplecontroller/v1alpha1.FooStatus)(0xc0001163a0),},
]
(dlv) p items[0].TypeMeta
k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.TypeMeta {
        Kind: "Foo",
        APIVersion: "samplecontroller.k8s.io/v1alpha1",}
(dlv) p items[0].ObjectMeta
k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.ObjectMeta {
        Name: "example-foo",
        GenerateName: "",
        Namespace: "default",
        SelfLink: "/apis/samplecontroller.k8s.io/v1alpha1/namespaces/default/foos/e...+10 more",
        UID: "118da85e-00fa-11e9-96e2-fa163e199d30",
        ResourceVersion: "204818",
        Generation: 5,
        CreationTimestamp: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.Time {
                Time: (*time.Time)(0xc000116308),},
        DeletionTimestamp: *k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.Time nil,
        DeletionGracePeriodSeconds: *int64 nil,
        Labels: map[string]string nil,
        Annotations: map[string]string nil,
        OwnerReferences: []k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.OwnerReference len: 0, cap: 0, nil,
        Initializers: *k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.Initializers nil,
        Finalizers: []string len: 0, cap: 0, nil,
        ClusterName: "",}
(dlv) p items[0].Spec
k8s.io/sample-controller/pkg/apis/samplecontroller/v1alpha1.FooSpec {
        DeploymentName: "example-foo",
        Replicas: *1,}
(dlv) p items[0].Status
k8s.io/sample-controller/pkg/apis/samplecontroller/v1alpha1.FooStatus {AvailableReplicas: 1}

再watch

在ListAndWatch中实现了 watchHandler方法（图中 2 Add Objects）， watchHandler会将数据存放在store中（DeltaFIFO）。

图中 1) watch

申请client request.watch的时候，会生成一个watch对象， 并启动receive

// NewStreamWatcher creates a StreamWatcher from the given decoder.
func NewStreamWatcher(d Decoder) *StreamWatcher {
        sw := &StreamWatcher{
                source: d,
                // It's easy for a consumer to add buffering via an extra
                // goroutine/channel, but impossible for them to remove it,
                // so nonbuffered is better.
                result: make(chan Event),
        }
        go sw.receive()
        return sw
}

　　

(dlv) p r.store
k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.Store(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.DeltaFIFO) *{
        lock: sync.RWMutex {
                w: (*sync.Mutex)(0xc000124630),
                writerSem: 0,
                readerSem: 0,
                readerCount: 0,
                readerWait: 0,},
        cond: sync.Cond {
                noCopy: sync.noCopy {},
                L: sync.Locker(*sync.RWMutex) ...,
                notify: (*sync.notifyList)(0xc000124658),
                checker: 824634918520,},
        items: map[string]k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.Deltas [],
        queue: []string len: 0, cap: 0, [],
        populated: true,
        initialPopulationCount: 0,
        keyFunc: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.MetaNamespaceKeyFunc,
        knownObjects: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.KeyListerGetter(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.cache) *{
                cacheStorage: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ThreadSafeStore(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.threadSafeMap) ...,
                keyFunc: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.DeletionHandlingMetaNamespaceKeyFunc,},
        closed: false,
        closedLock: sync.Mutex {state: 0, sema: 0},}

watch的定义如下：

(dlv) p w
k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/watch.Interface(*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/watch.StreamWatcher) *{
        Mutex: sync.Mutex {state: 0, sema: 0},
        source: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/watch.Decoder(*k8s.io/sample-controller/vendor/k8s.io/client-go/rest/watch.Decoder) *{
                decoder: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming.Decoder(*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming.decoder) ...,
                embeddedDecoder: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/runtime.Decoder(k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.DirectDecoder) *(*"k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/runtime.Decoder")(0xc00030c6f0),},
        result: chan k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/watch.Event {
                qcount: 0,
                dataqsiz: 0,
                buf: *[0]struct k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/watch.Event [],
                elemsize: 32,
                closed: 1,
                elemtype: *runtime._type {
                        size: 32,
                        ptrdata: 32,
                        hash: 107209836,
                        tflag: tflagUncommon|tflagExtraStar|tflagNamed,
                        align: 8,
                        fieldalign: 8,
                        kind: 25,
                        alg: *(*runtime.typeAlg)(0xf5ca70),
                        gcdata: *9,
                        str: 66689,
                        ptrToThis: 1120544,},
                sendx: 0,
                recvx: 0,
                recvq: waitq<k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/watch.Event> {
                        first: *sudog<k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/watch.Event> nil,
                        last: *sudog<k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/watch.Event> nil,},
                sendq: waitq<k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/watch.Event> {
                        first: *sudog<k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/watch.Event> nil,
                        last: *sudog<k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/watch.Event> nil,},
                lock: runtime.mutex {key: 0},},
        stopped: true,}

2. Informer

controller 初始化informer 的lister的时候，进行了先初始化了informer。此后加AddEventHandler

对于deployment informer 来说

k8s.io/sample-controller/vendor/k8s.io/client-go/informers.(*sharedInformerFactory).InformerFor() ./vendor/k8s.io/client-go/informers/factory.go:177

 bt
0  0x0000000000efbd8f in k8s.io/sample-controller/vendor/k8s.io/client-go/informers.(*sharedInformerFactory).InformerFor
   at ./vendor/k8s.io/client-go/informers/factory.go:177
1  0x0000000000e885f8 in k8s.io/sample-controller/vendor/k8s.io/client-go/informers/apps/v1.(*deploymentInformer).Informer
   at ./vendor/k8s.io/client-go/informers/apps/v1/deployment.go:84
2  0x0000000000e8866f in k8s.io/sample-controller/vendor/k8s.io/client-go/informers/apps/v1.(*deploymentInformer).Lister
   at ./vendor/k8s.io/client-go/informers/apps/v1/deployment.go:88
3  0x0000000000f44e5d in main.NewController
   at ./controller.go:107
4  0x0000000000f4742a in main.main
   at ./main.go:64
5  0x000000000042dfe7 in runtime.main
   at /usr/local/go/src/runtime/proc.go:201
6  0x00000000004594e1 in runtime.goexit
   at /usr/local/go/src/runtime/asm_amd64.s:1333

　　

 p newFunc
k8s.io/sample-controller/vendor/k8s.io/client-go/informers/apps/v1.(*deploymentInformer).defaultInformer-fm

　　

p informer
k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.SharedIndexInformer(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.sharedIndexInformer) *{
        indexer: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.Indexer(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.cache) *{
                cacheStorage: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ThreadSafeStore(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.threadSafeMap) ...,
                keyFunc: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.DeletionHandlingMetaNamespaceKeyFunc,},
        controller: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.Controller nil,
        processor: *k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.sharedProcessor {
                listenersStarted: false,
                listenersLock: (*sync.RWMutex)(0xc000315184),
                listeners: []*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.processorListener len: 0, cap: 0, nil,
                syncingListeners: []*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.processorListener len: 0, cap: 0, nil,
                clock: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.Clock(*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.RealClock) ...,
                wg: (*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.Group)(0xc0003151e0),},
        cacheMutationDetector: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.CacheMutationDetector(k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.dummyMutationDetector) {},
        listerWatcher: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ListerWatcher(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ListWatch) *{ListFunc: k8s.io/sample-controller/vendor/k8s.io/client-go/informers/apps/v1.NewFilteredDeploymentInformer.func1, WatchFunc: k8s.io/sample-controller/vendor/k8s.io/client-go/informers/apps/v1.NewFilteredDeploymentInformer.func2, DisableChunking: false},
        objectType: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/runtime.Object(*k8s.io/sample-controller/vendor/k8s.io/api/apps/v1.Deployment) *{
                TypeMeta: (*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.TypeMeta)(0xc00030cd80),
                ObjectMeta: (*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.ObjectMeta)(0xc00030cda0),
                Spec: (*k8s.io/sample-controller/vendor/k8s.io/api/apps/v1.DeploymentSpec)(0xc00030ce88),
                Status: (*k8s.io/sample-controller/vendor/k8s.io/api/apps/v1.DeploymentStatus)(0xc00030d150),},
        resyncCheckPeriod: k8s.io/sample-controller/vendor/k8s.io/klog.flushInterval,
        defaultEventHandlerResyncPeriod: k8s.io/sample-controller/vendor/k8s.io/klog.flushInterval,
        clock: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.Clock(*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.RealClock) *{},
        started: false,
        stopped: false,
        startedLock: sync.Mutex {state: 0, sema: 0},
        blockDeltas: sync.Mutex {state: 0, sema: 0},}

　　

(dlv) p informer.indexer
k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.Indexer(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.cache) *{
        cacheStorage: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ThreadSafeStore(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.threadSafeMap) *{
                lock: (*sync.RWMutex)(0xc000338f00),
                items: map[string]interface {} [],
                indexers: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.Indexers [...],
                indices: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.Indices [],},
        keyFunc: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.DeletionHandlingMetaNamespaceKeyFunc,}

deploymentInformer.Informer().HasSynced 会再一次InformerFor

(dlv) bt
0  0x0000000000efbca0 in k8s.io/sample-controller/vendor/k8s.io/client-go/informers.(*sharedInformerFactory).InformerFor
   at ./vendor/k8s.io/client-go/informers/factory.go:166
1  0x0000000000e885f8 in k8s.io/sample-controller/vendor/k8s.io/client-go/informers/apps/v1.(*deploymentInformer).Informer
   at ./vendor/k8s.io/client-go/informers/apps/v1/deployment.go:84
2  0x0000000000f44e8e in main.NewController
   at ./controller.go:108
3  0x0000000000f4742a in main.main
   at ./main.go:64
4  0x000000000042dfe7 in runtime.main
   at /usr/local/go/src/runtime/proc.go:201
5  0x00000000004594e1 in runtime.goexit
   at /usr/local/go/src/runtime/asm_amd64.s:1333

　　

(dlv) p f
*k8s.io/sample-controller/vendor/k8s.io/client-go/informers.sharedInformerFactory {
        client: k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes.Interface(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes.Clientset) *{
                DiscoveryClient: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/discovery.DiscoveryClient)(0xc00031f8e0),
                admissionregistrationV1alpha1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/admissionregistration/v1alpha1.AdmissionregistrationV1alpha1Client)(0xc0002fce50),
                admissionregistrationV1beta1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/admissionregistration/v1beta1.AdmissionregistrationV1beta1Client)(0xc0002fcec0),
                appsV1beta1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/apps/v1beta1.AppsV1beta1Client)(0xc0002fcf30),
                appsV1beta2: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/apps/v1beta2.AppsV1beta2Client)(0xc0002fcfa0),
                appsV1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/apps/v1.AppsV1Client)(0xc0002fd010),
                auditregistrationV1alpha1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/auditregistration/v1alpha1.AuditregistrationV1alpha1Client)(0xc0002fd080),
                authenticationV1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/authentication/v1.AuthenticationV1Client)(0xc0002fd0f0),
                authenticationV1beta1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/authentication/v1beta1.AuthenticationV1beta1Client)(0xc0002fd160),
                authorizationV1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/authorization/v1.AuthorizationV1Client)(0xc0002fd1d0),
                authorizationV1beta1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/authorization/v1beta1.AuthorizationV1beta1Client)(0xc0002fd240),
                autoscalingV1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/autoscaling/v1.AutoscalingV1Client)(0xc0002fd2b0),
                autoscalingV2beta1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/autoscaling/v2beta1.AutoscalingV2beta1Client)(0xc0002fd320),
                autoscalingV2beta2: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/autoscaling/v2beta2.AutoscalingV2beta2Client)(0xc0002fd390),
                batchV1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/batch/v1.BatchV1Client)(0xc0002fd400),
                batchV1beta1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/batch/v1beta1.BatchV1beta1Client)(0xc0002fd470),
                batchV2alpha1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/batch/v2alpha1.BatchV2alpha1Client)(0xc0002fd4e0),
                certificatesV1beta1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/certificates/v1beta1.CertificatesV1beta1Client)(0xc0002fd550),
                coordinationV1beta1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/coordination/v1beta1.CoordinationV1beta1Client)(0xc0002fd5c0),
                coreV1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/core/v1.CoreV1Client)(0xc0002fd630),
                eventsV1beta1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/events/v1beta1.EventsV1beta1Client)(0xc0002fd6a0),
                extensionsV1beta1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/extensions/v1beta1.ExtensionsV1beta1Client)(0xc0002fd710),
                networkingV1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/networking/v1.NetworkingV1Client)(0xc0002fd780),
                policyV1beta1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/policy/v1beta1.PolicyV1beta1Client)(0xc0002fd7f0),
                rbacV1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/rbac/v1.RbacV1Client)(0xc0002fd860),
                rbacV1beta1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/rbac/v1beta1.RbacV1beta1Client)(0xc0002fd8d0),
                rbacV1alpha1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/rbac/v1alpha1.RbacV1alpha1Client)(0xc0002fd940),
                schedulingV1alpha1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/scheduling/v1alpha1.SchedulingV1alpha1Client)(0xc0002fd9b0),
                schedulingV1beta1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/scheduling/v1beta1.SchedulingV1beta1Client)(0xc0002fda20),
                settingsV1alpha1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/settings/v1alpha1.SettingsV1alpha1Client)(0xc0002fda90),
                storageV1beta1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/storage/v1beta1.StorageV1beta1Client)(0xc0002fdb00),
                storageV1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/storage/v1.StorageV1Client)(0xc0002fdb70),
                storageV1alpha1: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/kubernetes/typed/storage/v1alpha1.StorageV1alpha1Client)(0xc0002fdbe0),},
        namespace: "",
        tweakListOptions: nil,
        lock: sync.Mutex {state: 1, sema: 0},
        defaultResync: k8s.io/sample-controller/vendor/k8s.io/klog.flushInterval,
        customResync: map[reflect.Type]time.Duration [],
        informers: map[reflect.Type]k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.SharedIndexInformer [
                ...: ...,
        ],
        startedInformers: map[reflect.Type]bool [],}
(dlv) p f.informers
map[reflect.Type]k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.SharedIndexInformer [
        *reflect.rtype {size: 8, ptrdata: 8, hash: 3543491241, tflag: tflagUncommon, align: 8, fieldAlign: 8, kind: 54, alg: *(*reflect.typeAlg)(0x1ca5ef0), gcdata: *1, str: 106883, ptrToThis: 0}: *k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.sharedIndexInformer {
                indexer: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.Indexer(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.cache) ...,
                controller: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.Controller nil,
                processor: *(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.sharedProcessor)(0xc000315180),
                cacheMutationDetector: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.CacheMutationDetector(k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.dummyMutationDetector) *(*"k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.CacheMutationDetector")(0xc000382028),
                listerWatcher: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ListerWatcher(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ListWatch) ...,
                objectType: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/runtime.Object(*k8s.io/sample-controller/vendor/k8s.io/api/apps/v1.Deployment) ...,
                resyncCheckPeriod: k8s.io/sample-controller/vendor/k8s.io/klog.flushInterval,
                defaultEventHandlerResyncPeriod: k8s.io/sample-controller/vendor/k8s.io/klog.flushInterval,
                clock: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.Clock(*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.RealClock) ...,
                started: false,
                stopped: false,
                startedLock: (*sync.Mutex)(0xc00038207c),
                blockDeltas: (*sync.Mutex)(0xc000382084),},
]

fooInformer

(dlv) p informer
k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.SharedIndexInformer(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.sharedIndexInformer) *{
        indexer: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.Indexer(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.cache) *{
                cacheStorage: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ThreadSafeStore(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.threadSafeMap) ...,
                keyFunc: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.DeletionHandlingMetaNamespaceKeyFunc,},
        controller: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.Controller nil,
        processor: *k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.sharedProcessor {
                listenersStarted: false,
                listenersLock: (*sync.RWMutex)(0xc0003151f4),
                listeners: []*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.processorListener len: 0, cap: 0, nil,
                syncingListeners: []*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.processorListener len: 0, cap: 0, nil,
                clock: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.Clock(*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.RealClock) ...,
                wg: (*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.Group)(0xc000315250),},
        cacheMutationDetector: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.CacheMutationDetector(k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.dummyMutationDetector) {},
        listerWatcher: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ListerWatcher(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ListWatch) *{ListFunc: k8s.io/sample-controller/pkg/client/informers/externalversions/samplecontroller/v1alpha1.NewFilteredFooInformer.func1, WatchFunc: k8s.io/sample-controller/pkg/client/informers/externalversions/samplecontroller/v1alpha1.NewFilteredFooInformer.func2, DisableChunking: false},
        objectType: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/runtime.Object(*k8s.io/sample-controller/pkg/apis/samplecontroller/v1alpha1.Foo) *{
                TypeMeta: (*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.TypeMeta)(0xc000346c80),
                ObjectMeta: (*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.ObjectMeta)(0xc000346ca0),
                Spec: (*k8s.io/sample-controller/pkg/apis/samplecontroller/v1alpha1.FooSpec)(0xc000346d88),
                Status: (*k8s.io/sample-controller/pkg/apis/samplecontroller/v1alpha1.FooStatus)(0xc000346da0),},
        resyncCheckPeriod: k8s.io/sample-controller/vendor/k8s.io/klog.flushInterval,
        defaultEventHandlerResyncPeriod: k8s.io/sample-controller/vendor/k8s.io/klog.flushInterval,
        clock: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.Clock(*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.RealClock) *{},
        started: false,
        stopped: false,
        startedLock: sync.Mutex {state: 0, sema: 0},
        blockDeltas: sync.Mutex {state: 0, sema: 0},}
(dlv) p informer.indexer
k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.Indexer(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.cache) *{
        cacheStorage: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ThreadSafeStore(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.threadSafeMap) *{
                lock: (*sync.RWMutex)(0xc000339020),
                items: map[string]interface {} [],
                indexers: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.Indexers [...],
                indices: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.Indices [],},
        keyFunc: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.DeletionHandlingMetaNamespaceKeyFunc,}
(dlv) p informer.indexer.cacheStorage
k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ThreadSafeStore(*k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.threadSafeMap) *{
        lock: sync.RWMutex {
                w: (*sync.Mutex)(0xc000339020),
                writerSem: 0,
                readerSem: 0,
                readerCount: 0,
                readerWait: 0,},
        items: map[string]interface {} [],
        indexers: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.Indexers [
                "namespace": k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.MetaNamespaceIndexFunc,
        ],
        indices: k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.Indices [],}

　　

k8s.io/sample-controller/pkg/client/informers/externalversions/samplecontroller/v1alpha1.(*fooInformer).defaultInformer-fm

func (f *sharedInformerFactory) InformerFor(obj runtime.Object, newFunc internalinterfaces.NewInformerFunc) cache.SharedIndexInformer {
        f.lock.Lock()
        defer f.lock.Unlock()

        informerType := reflect.TypeOf(obj)
        informer, exists := f.informers[informerType]
        if exists {
                return informer
        }

        resyncPeriod, exists := f.customResync[informerType]
        if !exists {
                resyncPeriod = f.defaultResync
        }

        informer = newFunc(f.client, resyncPeriod)
        f.informers[informerType] = informer

        return informer
}

(dlv) p newFunc
k8s.io/sample-controller/pkg/client/informers/externalversions/samplecontroller/v1alpha1.(*fooInformer).defaultInformer-fm

(dlv) p obj
k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/runtime.Object(*k8s.io/sample-controller/pkg/apis/samplecontroller/v1alpha1.Foo) *{
        TypeMeta: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.TypeMeta {Kind: "", APIVersion: ""},
        ObjectMeta: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.ObjectMeta {
                Name: "",
                GenerateName: "",
                Namespace: "",
                SelfLink: "",
                UID: "",
                ResourceVersion: "",
                Generation: 0,
                CreationTimestamp: (*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.Time)(0xc000346bc8),
                DeletionTimestamp: *k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.Time nil,
                DeletionGracePeriodSeconds: *int64 nil,
                Labels: map[string]string nil,
                Annotations: map[string]string nil,
                OwnerReferences: []k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.OwnerReference len: 0, cap: 0, nil,
                Initializers: *k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.Initializers nil,
                Finalizers: []string len: 0, cap: 0, nil,
                ClusterName: "",},
        Spec: k8s.io/sample-controller/pkg/apis/samplecontroller/v1alpha1.FooSpec {DeploymentName: "", Replicas: *int32 nil},
        Status: k8s.io/sample-controller/pkg/apis/samplecontroller/v1alpha1.FooStatus {AvailableReplicas: 0},}
(dlv) ls

　　

(dlv) s
> k8s.io/sample-controller/pkg/client/informers/externalversions/samplecontroller/v1alpha1.(*fooInformer).defaultInformer-fm() ./pkg/client/informers/externalversions/samplecontroller/v1alpha1/foo.go:84 (PC: 0xea835f)
Warning: debugging optimized function
    79: func (f *fooInformer) defaultInformer(client versioned.Interface, resyncPeriod time.Duration) cache.SharedIndexInformer {
    80:         return NewFilteredFooInformer(client, f.namespace, resyncPeriod, cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}, f.tweakListOptions)
    81: }
    82:
    83: func (f *fooInformer) Informer() cache.SharedIndexInformer {
=>  84:         return f.factory.InformerFor(&samplecontrollerv1alpha1.Foo{}, f.defaultInformer)
    85: }
    86:
    87: func (f *fooInformer) Lister() v1alpha1.FooLister {
    88:         return v1alpha1.NewFooLister(f.Informer().GetIndexer())
    89: }
(dlv) bt
0  0x0000000000ea835f in k8s.io/sample-controller/pkg/client/informers/externalversions/samplecontroller/v1alpha1.(*fooInformer).defaultInformer-fm
   at ./pkg/client/informers/externalversions/samplecontroller/v1alpha1/foo.go:84
1  0x0000000000f43439 in k8s.io/sample-controller/pkg/client/informers/externalversions.(*sharedInformerFactory).InformerFor
   at ./pkg/client/informers/externalversions/factory.go:162
2  0x0000000000ea7ce8 in k8s.io/sample-controller/pkg/client/informers/externalversions/samplecontroller/v1alpha1.(*fooInformer).Informer
   at ./pkg/client/informers/externalversions/samplecontroller/v1alpha1/foo.go:84
3  0x0000000000ea7d5f in k8s.io/sample-controller/pkg/client/informers/externalversions/samplecontroller/v1alpha1.(*fooInformer).Lister
   at ./pkg/client/informers/externalversions/samplecontroller/v1alpha1/foo.go:88
4  0x0000000000f44ebf in main.NewController
   at ./controller.go:109
5  0x0000000000f4742a in main.main
   at ./main.go:64
6  0x000000000042dfe7 in runtime.main
   at /usr/local/go/src/runtime/proc.go:201
7  0x00000000004594e1 in runtime.goexit
   at /usr/local/go/src/runtime/asm_amd64.s:1333

workqueue

(dlv) p q
*k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.delayingType {
        Interface: k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.Interface(*k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.Type) *{
                queue: []k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.t len: 0, cap: 0, [],
                dirty: k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.set [],
                processing: k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.set [],
                cond: *(*sync.Cond)(0xc0003c2040),
                shuttingDown: false,
                metrics: k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.queueMetrics(k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.noMetrics) *(*"k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.queueMetrics")(0xc00035f0b8),
                unfinishedWorkUpdatePeriod: net/http.rstAvoidanceDelay,
                clock: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.Clock(k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.RealClock) *(*"k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.Clock")(0xc00035f0d0),},
        clock: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.Clock(k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.RealClock) {},
        stopCh: chan struct {} {
                qcount: 0,
                dataqsiz: 0,
                buf: *[0]struct struct {} [],
                elemsize: 0,
                closed: 0,
                elemtype: *runtime._type {
                        size: 0,
                        ptrdata: 0,
                        hash: 670477339,
                        tflag: tflagExtraStar,
                        align: 1,
                        fieldalign: 1,
                        kind: 153,
                        alg: *(*runtime.typeAlg)(0x1ca5eb0),
                        gcdata: *1,
                        str: 47987,
                        ptrToThis: 496000,},
                sendx: 0,
                recvx: 0,
                recvq: waitq<struct {}> {
                        first: *(*sudog<struct {}>)(0xc0002e8120),
                        last: *(*sudog<struct {}>)(0xc0002e8120),},
                sendq: waitq<struct {}> {
                        first: *sudog<struct {}> nil,
                        last: *sudog<struct {}> nil,},
                lock: runtime.mutex {key: 0},},
        heartbeat: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.Ticker(*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.realTicker) *{
                ticker: *(*time.Ticker)(0xc000394aa0),},
        waitingForAddCh: chan *k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.waitFor {
                qcount: 0,
                dataqsiz: 1000,
                buf: *[1000]*struct k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.waitFor [
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        *nil,
                        ...+936 more
                ],
                elemsize: 8,
                closed: 0,
                elemtype: *runtime._type {
                        size: 8,
                        ptrdata: 8,
                        hash: 3317760887,
                        tflag: 0,
                        align: 8,
                        fieldalign: 8,
                        kind: 54,
                        alg: *(*runtime.typeAlg)(0x1ca5ef0),
                        gcdata: *1,
                        str: 145247,
                        ptrToThis: 0,},
                sendx: 3,
                recvx: 3,
                recvq: waitq<*k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.waitFor> {
                        first: *(*sudog<*k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.waitFor>)(0xc0002e81e0),
                        last: *(*sudog<*k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.waitFor>)(0xc0002e81e0),},
                sendq: waitq<*k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.waitFor> {
                        first: *sudog<*k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.waitFor> nil,
                        last: *sudog<*k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.waitFor> nil,},
                lock: runtime.mutex {key: 0},},
        metrics: k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.retryMetrics(*k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.defaultRetryMetrics) *{
                retries: k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.CounterMetric(k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.noopMetric) *(*"k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.CounterMetric")(0xc00038eba0),},}
(dlv) p q.Interface
k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.Interface(*k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.Type) *{
        queue: []k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.t len: 0, cap: 0, [],
        dirty: k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.set [],
        processing: k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.set [],
        cond: *sync.Cond {
                noCopy: sync.noCopy {},
                L: sync.Locker(*sync.Mutex) ...,
                notify: (*sync.notifyList)(0xc0003c2050),
                checker: 824637661296,},
        shuttingDown: false,
        metrics: k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.queueMetrics(k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.noMetrics) {},
        unfinishedWorkUpdatePeriod: net/http.rstAvoidanceDelay,
        clock: k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.Clock(k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/clock.RealClock) {},}

　　

    43:
    44: func NewNamedRateLimitingQueue(rateLimiter RateLimiter, name string) RateLimitingInterface {
    45:         return &rateLimitingType{
=>  46:                 DelayingInterface: NewNamedDelayingQueue(name),
    47:                 rateLimiter:       rateLimiter,
    48:         }
    49: }
    50:
    51: // rateLimitingType wraps an Interface and provides rateLimited re-enquing
(dlv) bt
0  0x0000000000ea12a1 in k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.NewNamedRateLimitingQueue
   at ./vendor/k8s.io/client-go/util/workqueue/rate_limitting_queue.go:46
1  0x0000000000f45087 in main.NewController
   at ./controller.go:111
2  0x0000000000f4742a in main.main
   at ./main.go:64
3  0x000000000042dfe7 in runtime.main
   at /usr/local/go/src/runtime/proc.go:201
4  0x00000000004594e1 in runtime.goexit
   at /usr/local/go/src/runtime/asm_amd64.s:1333

　　

sharedIndexInformer.Run 会注册 HandleDeltas， 放在defaultCacheMutationDetector.cachedObjs中，defaultCacheMutationDetector.Run 会CompareObjects. 这个在sample的例子中貌似没有用到。 

在"vendor/k8s.io/client-go/tools/cache/controller.go"的NewInformer或NewIndexerInformer的时候，也会重新实现Process， 做Sync, Added, Updated，和Deleted， 放在store（NewInformer）或者NewIndexer（NewIndexerInformer）中。processLoop  会排干这个queue.

// processLoop drains the work queue.
// TODO: Consider doing the processing in parallel. This will require a little thought
// to make sure that we don't end up processing the same object multiple times
// concurrently.
//
// TODO: Plumb through the stopCh here (and down to the queue) so that this can
// actually exit when the controller is stopped. Or just give up on this stuff
// ever being stoppable. Converting this whole package to use Context would
// also be helpful.
func (c *controller) processLoop() {
        for {
                obj, err := c.config.Queue.Pop(PopProcessFunc(c.config.Process))
                if err != nil {
                        if err == FIFOClosedError {
                                return
                        }
                        if c.config.RetryOnError {
                                // This is the safe way to re-enqueue.
                                c.config.Queue.AddIfNotPresent(obj)
                        }
                }
        }
}

　　

func (s *sharedIndexInformer) HandleDeltas(obj interface{}) error {
        s.blockDeltas.Lock()
        defer s.blockDeltas.Unlock()

        // from oldest to newest
        for _, d := range obj.(Deltas) {
                switch d.Type {
                case Sync, Added, Updated:
                        isSync := d.Type == Sync
                        s.cacheMutationDetector.AddObject(d.Object)
                        if old, exists, err := s.indexer.Get(d.Object); err == nil && exists {
                                if err := s.indexer.Update(d.Object); err != nil {
                                        return err
                                }
                                s.processor.distribute(updateNotification{oldObj: old, newObj: d.Object}, isSync)
                        } else {
                                if err := s.indexer.Add(d.Object); err != nil {
                                        return err
                                }
                                s.processor.distribute(addNotification{newObj: d.Object}, isSync)
                        }
                case Deleted:
                        if err := s.indexer.Delete(d.Object); err != nil {
                                return err
                        }
                        s.processor.distribute(deleteNotification{oldObj: d.Object}, false)
                }
        }
        return nil
}


func (s *sharedIndexInformer) Run(stopCh <-chan struct{}) {
        defer utilruntime.HandleCrash()

        fifo := NewDeltaFIFO(MetaNamespaceKeyFunc, s.indexer)

        cfg := &Config{
                Queue:            fifo,
                ListerWatcher:    s.listerWatcher,
                ObjectType:       s.objectType,
                FullResyncPeriod: s.resyncCheckPeriod,
                RetryOnError:     false,
                ShouldResync:     s.processor.shouldResync,

                Process: s.HandleDeltas,
        }

        func() {
                s.startedLock.Lock()
                defer s.startedLock.Unlock()

                s.controller = New(cfg)
                s.controller.(*controller).clock = s.clock
                s.started = true
        }()

        // Separate stop channel because Processor should be stopped strictly after controller
        processorStopCh := make(chan struct{})
        var wg wait.Group
        defer wg.Wait()              // Wait for Processor to stop
        defer close(processorStopCh) // Tell Processor to stop
        wg.StartWithChannel(processorStopCh, s.cacheMutationDetector.Run)
        wg.StartWithChannel(processorStopCh, s.processor.run)

        defer func() {
                s.startedLock.Lock()
                defer s.startedLock.Unlock()
                s.stopped = true // Don't want any new listeners
        }()
        s.controller.Run(stopCh)
}

　　

(dlv) b sharedInformerFactory.Start
(dlv) b k8s.io/sample-controller/vendor/k8s.io/client-go/informers.(*sharedInformerFactory).Start
(dlv) b k8s.io/sample-controller/pkg/client/informers/externalversions.(*sharedInformerFactory).Start
(dlv) b sharedIndexInformer.Run
(dlv) b defaultCacheMutationDetector.Run
(dlv) b sharedProcessor.run
(dlv) bp
Breakpoint b at 0xf42f08 for k8s.io/sample-controller/pkg/client/informers/externalversions.(*sharedInformerFactory).Start() ./pkg/client/informers/externalversions/factory.go:111 (1)
Breakpoint 2 at 0xe77f98 for k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run() ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:189 (1)
Breakpoint 3 at 0xe74128 for k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*defaultCacheMutationDetector).Run() ./vendor/k8s.io/client-go/tools/cache/mutation_detector.go:82 (0)
Breakpoint 4 set at 0xe79b13 for k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*sharedProcessor).run() ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:415

　　

图中 3->4->5的流程

(dlv) bt
 0  0x0000000000e7b9b3 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*threadSafeMap).Add
    at ./vendor/k8s.io/client-go/tools/cache/thread_safe_store.go:68
 1  0x0000000000e7ae92 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*cache).Add
    at ./vendor/k8s.io/client-go/tools/cache/store.go:128
 2  0x0000000000e79276 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).HandleDeltas
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:360
 3  0x0000000000e7f40e in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).HandleDeltas-fm
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:202
 4  0x0000000000e71afd in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop
    at ./vendor/k8s.io/client-go/tools/cache/delta_fifo.go:436
 5  0x0000000000e6f7f0 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop
    at ./vendor/k8s.io/client-go/tools/cache/controller.go:150
 6  0x0000000000e7f31a in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop-fm
    at ./vendor/k8s.io/client-go/tools/cache/controller.go:124
 7  0x00000000009b4c14 in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
 8  0x00000000009b451e in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
 9  0x00000000009b444d in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.Until
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
10  0x0000000000e6f639 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*controller).Run
    at ./vendor/k8s.io/client-go/tools/cache/controller.go:124
11  0x0000000000e7838d in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:227
12  0x00000000004594e1 in runtime.goexit
    at /usr/local/go/src/runtime/asm_amd64.s:1333

图中 6->7的流程

(dlv) bt
 0  0x0000000000f46733 in main.(*Controller).enqueueFoo
    at ./controller.go:338
 1  0x0000000000f4785e in main.NewController.func1
    at ./controller.go:120
 2  0x0000000000e6f96d in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate
    at ./vendor/k8s.io/client-go/tools/cache/controller.go:202
 3  0x0000000000e81066 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnUpdate
    at <autogenerated>:1
 4  0x0000000000e7ee3b in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1.1
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:552
 5  0x00000000009b478c in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203
 6  0x0000000000e7efa9 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:548
 7  0x00000000009b4c14 in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
 8  0x00000000009b451e in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
 9  0x00000000009b444d in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.Until
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
10  0x0000000000e7a58d in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:546
11  0x0000000000e7f50a in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run-fm
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:390
12  0x00000000009b4b8f in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
13  0x00000000004594e1 in runtime.goexit
    at /usr/local/go/src/runtime/asm_amd64.s:1333

　

 0  0x0000000000e9ed61 in k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.(*delayingType).AddAfter
    at ./vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:159
 1  0x0000000000ea13e2 in k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.(*rateLimitingType).AddRateLimited
    at ./vendor/k8s.io/client-go/util/workqueue/rate_limitting_queue.go:60
 2  0x0000000000f467ed in main.(*Controller).enqueueFoo
    at ./controller.go:345
 3  0x0000000000f46b2e in main.(*Controller).handleObject
    at ./controller.go:383
 4  0x0000000000f47f5e in main.(*Controller).handleObject-fm
    at ./controller.go:130
 5  0x0000000000e6f8f9 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd
    at ./vendor/k8s.io/client-go/tools/cache/controller.go:195
 6  0x0000000000e80f32 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnAdd
    at <autogenerated>:1
 7  0x0000000000e7eecd in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1.1
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:554
 8  0x00000000009b478c in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203
 9  0x0000000000e7efa9 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:548
10  0x00000000009b4c14 in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
11  0x00000000009b451e in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
12  0x00000000009b444d in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.Until
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
13  0x0000000000e7a58d in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:546
14  0x0000000000e7f50a in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run-fm
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:390
15  0x00000000009b4b8f in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
16  0x00000000004594e1 in runtime.goexit
    at /usr/local/go/src/runtime/asm_amd64.s:1333

图中8->9

0  0x0000000000ea0c8b in k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.(*Type).Get
   at ./vendor/k8s.io/client-go/util/workqueue/queue.go:152
1  0x0000000000ea188c in k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.(*delayingType).Get
   at <autogenerated>:1
2  0x0000000000ea1d8c in k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.(*rateLimitingType).Get
   at <autogenerated>:1
3  0x0000000000f45b98 in main.(*Controller).processNextWorkItem
   at ./controller.go:188
4  0x0000000000f45b3b in main.(*Controller).runWorker
   at ./controller.go:181
5  0x0000000000f47f9a in main.(*Controller).runWorker-fm
   at ./controller.go:167
6  0x00000000009b4c14 in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
   at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
7  0x00000000009b451e in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil
   at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
8  0x00000000009b444d in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.Until
   at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
9  0x00000000004594e1 in runtime.goexit
   at /usr/local/go/src/runtime/asm_amd64.s:1333

REF：  

深入浅出kubernetes之client-go     

深入浅出kubernetes之client-go的Indexer

Kubernetes的client-go库介绍  (tencent)

Kubernetes Client-Go Informer 实现源码剖析

kube-controller-manager源码分析（三）之 Informer机制

kubernetes进阶：informer机制解析  

go-client Indexer   

Kubernetes编写自定义controller   

kubernetes client-go解析  （推荐）

github的API 相关：  

Kubernetes源码详解

kubernetes源码分析  

Learning k8s source code