目录
operator-sdk 环境搭建与简单实例
环境说明
系统：CentOS Linux release 7.5.1804 (Core)
golang：v1.15
operator-sdk：v1.7.0
docker：v1.19 # 因为 operator-sdk 使用了多阶段构建功能，所以 docker 必须大于等于 v1.17 版本
k8s：k3s v1.20.6+k3s1 单节点
安装 docker
略
安装 golang
安装
$ yum install epel-release -y
$ yum install golang -y
配置
$ mkdir -p /home/workspace/go/{src,bin,pkg}
$ vi /etc/profile
export GOPATH=/home/workspace/go
export GO111MODULE=on
export GOPROXY=https://goproxy.io,direct
export PATH=$PATH:$GOPATH/bin
$ . /etc/profile
验证
$ go version
go version go1.15.5 linux/amd64
安装 operator-sdk
operator-sdk 为 go 语言编写，直接下载二进制执行文件安装即可，下载地址：
https://github.com/operator-framework/operator-sdk/releases
安装步骤略。
创建一个简单实例
步骤1: init操作
$ mkdir -p $GOPATH/src/github.com/leffss/memcached-operator
$ cd $GOPATH/src/github.com/leffss/memcached-operator
$ operator-sdk init --domain=example.com --repo=github.com/leffss/memcached-operator
步骤2: 创建API
$ operator-sdk create api --group cache --version v1 --kind Memcached --resource=true --controller=true
步骤3：安装 CRDs
$ make generate && make manifests && make install
步骤4: 构建镜像
$ vi Dockerfile
# 在 RUN go mod download 前面新增 goproxy
ENV GOPROXY https://goproxy.io,direct
# 注释 FROM gcr.io/distroless/static:nonroot（国外源，pull 不下来），新增
FROM kubeimages/distroless-static:latest
$ make docker-build IMG=leffss/memcache:v1
$ docker images |grep memcache
leffss/memcache v1 1eaa4b6a4781 About a minute ago 46.5MB
- 如遇到包下载错误（网络问题），多尝试几次
步骤5: 运行Operator
准备镜像：
# 因为 k3s 安装的 k8s v1.20 默认使用的是 containerd，所以要导入镜像
$ docker save leffss/memcache:v1 > memcache.tar
$ ctr image import memcache.tar
$ docker pull kubesphere/kube-rbac-proxy:v0.8.0
$ docker tag kubesphere/kube-rbac-proxy:v0.8.0 gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0
$ docker save gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0 > kube-rabc-proxy.tar
$ ctr image import kube-rabc-proxy.tar
运行
$ make deploy IMG=leffss/memcache:v1
结果确认：
$ kubectl get service -A |grep memcache
memcached-operator-system memcached-operator-controller-manager-metrics-service ClusterIP 10.43.212.38 <none> 8443/TCP 83s
$ kubectl -n memcached-operator-system get pod
NAME READY STATUS RESTARTS AGE
memcached-operator-controller-manager-5cd7d4cdc7-tpsh2 2/2 Running 0 9m10s
$ kubectl -n memcached-operator-system get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
memcached-operator-controller-manager 1/1 1 1 26m
CRDs 确认
$ kubectl get crd
NAME CREATED AT
addons.k3s.cattle.io 2021-05-02T01:03:34Z
helmcharts.helm.cattle.io 2021-05-02T01:03:34Z
helmchartconfigs.helm.cattle.io 2021-05-02T01:03:34Z
memcacheds.cache.example.com 2021-05-02T06:55:23Z
详细确认
$ kubectl describe crd memcacheds.cache.example.com
Name: memcacheds.cache.example.com
Namespace:
Labels: <none>
Annotations: controller-gen.kubebuilder.io/version: v0.4.1
API Version: apiextensions.k8s.io/v1
Kind: CustomResourceDefinition
Metadata:
Creation Timestamp: 2021-05-02T06:55:23Z
Generation: 1
Managed Fields:
API Version: apiextensions.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:acceptedNames:
f:listKind:
f:singular:
Manager: k3s
Operation: Update
Time: 2021-05-02T06:55:23Z
API Version: apiextensions.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:controller-gen.kubebuilder.io/version:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
f:conversion:
.:
f:strategy:
f:group:
f:names:
f:kind:
f:listKind:
f:plural:
f:singular:
f:scope:
f:versions:
f:status:
f:acceptedNames:
f:kind:
f:plural:
f:conditions:
f:storedVersions:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2021-05-02T06:55:26Z
Resource Version: 16326
UID: 361420cd-5785-4751-ab40-bb0e4482d0cf
Spec:
Conversion:
Strategy: None
Group: cache.example.com
Names:
Kind: Memcached
List Kind: MemcachedList
Plural: memcacheds
Singular: memcached
Scope: Namespaced
Versions:
Name: v1
Schema:
openAPIV3Schema:
Description: Memcached is the Schema for the memcacheds API
Properties:
API Version:
Description: APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
Type: string
Kind:
Description: Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
Type: string
Metadata:
Type: object
Spec:
Description: MemcachedSpec defines the desired state of Memcached
Properties:
Foo:
Description: Foo is an example field of Memcached. Edit memcached_types.go to remove/update
Type: string
Type: object
Status:
Description: MemcachedStatus defines the observed state of Memcached
Type: object
Type: object
Served: true
Storage: true
Subresources:
Status:
Status:
Accepted Names:
Kind: Memcached
List Kind: MemcachedList
Plural: memcacheds
Singular: memcached
Conditions:
Last Transition Time: 2021-05-02T06:55:23Z
Message: no conflicts found
Reason: NoConflicts
Status: True
Type: NamesAccepted
Last Transition Time: 2021-05-02T06:55:23Z
Message: the initial names have been accepted
Reason: InitialNamesAccepted
Status: True
Type: Established
Stored Versions:
v1
Events: <none>
步骤6: 创建自定义资源
$ kubectl apply -f config/samples/cache_v1_memcached.yaml
memcached.cache.example.com/memcached-sample created
确认Operator相关日志信息
$ kubectl -n memcached-operator-system logs memcached-operator-controller-manager-6cf86db855-sqhpj -c manager
2021-05-02T07:11:32.316Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": "127.0.0.1:8080"}
2021-05-02T07:11:32.317Z INFO setup starting manager
I0502 07:11:32.317614 1 leaderelection.go:243] attempting to acquire leader lease memcached-operator-system/2ef0186a.example.com...
2021-05-02T07:11:32.317Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
I0502 07:11:32.414648 1 leaderelection.go:253] successfully acquired lease memcached-operator-system/2ef0186a.example.com
2021-05-02T07:11:32.415Z INFO controller-runtime.manager.controller.memcached Starting EventSource {"reconciler group": "cache.example.com", "reconciler kind": "Memcached", "source": "kind source: /, Kind="}
2021-05-02T07:11:32.415Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"memcached-operator-system","name":"2ef0186a.example.com","uid":"b8b753e8-6400-4adc-96bd-365ce8d9dabc","apiVersion":"v1","resourceVersion":"17936"}, "reason": "LeaderElection", "message": "memcached-operator-controller-manager-6cf86db855-sqhpj_ac983311-240b-47da-a7cd-2b8753c8fe21 became leader"}
2021-05-02T07:11:32.415Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"Lease","namespace":"memcached-operator-system","name":"2ef0186a.example.com","uid":"1ee59e1e-4481-4b84-a3bd-0f0b1e97979d","apiVersion":"coordination.k8s.io/v1","resourceVersion":"17937"}, "reason": "LeaderElection", "message": "memcached-operator-controller-manager-6cf86db855-sqhpj_ac983311-240b-47da-a7cd-2b8753c8fe21 became leader"}
2021-05-02T07:11:32.515Z INFO controller-runtime.manager.controller.memcached Starting Controller {"reconciler group": "cache.example.com", "reconciler kind": "Memcached"}
2021-05-02T07:11:32.515Z INFO controller-runtime.manager.controller.memcached Starting workers {"reconciler group": "cache.example.com", "reconciler kind": "Memcached", "worker count": 1}
创建 crd 资源后可能 controller 会包 RBAC 权限错误，解决方法是修改部署时的权限配置，最简单的方法是直接给 controller 绑定到 cluster-admin 集群管理员
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-admin-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: app-controller-manager
namespace: app-system
步骤7: 删除CR和相关资源
$ kubectl delete -f config/samples/cache_v1_memcached.yaml
memcached.cache.example.com "memcached-sample" deleted
步骤8：删除 CRDs
$ make uninstall
步骤9：删除 operator
$ make undeploy
总结
按照前面步骤的一系列的操作，我们使用了 sdk 创建的默认crd的属性和controller代码，实际开发中需要根据需求自行定义 crd 属性，并编写对应的 controller 逻辑代码，最终实现一个完整的 operator。
补充：kubebuilder和operator-sdk对比
kubebuilder和operator-sdk都是开发crd的工具。其中kubebuilder是k8s官方提供工具，operator-sdk早期版本使用了其自己的逻辑实现，但是目前这两个工具都是在controller-runtime基础上做了一层封装,底层逻辑代码基本没什么多大的区别，可以看个人习惯自行选择。