首页 > golang > k8s源码学习-三大核心数据结构
2019
08-16

k8s源码学习-三大核心数据结构

三大核心数据结构是什么?

Group(APIGroup) 资源组
Version(APIVersions) 资源版本
Resource/SubResource(APIResource) 资源/子资源
Kind 资源种类,描述Resource的种类

完整表现形式:资源组/资源版本/资源/子资源
<group>/<version>/<resource>/<subresource>
apps/v1/deployments/status

三大核心数据结构如何定义

以github当前最新版本Kubernetes v1.23.3为例:https://github.com/kubernetes/kubernetes

Group资源组

代码:vendor/k8s.io/apimachinery/pkg/apis/meta/v1/types.go

// APIGroup contains the name, the supported versions, and the preferred version
// of a group.
type APIGroup struct {
    TypeMeta `json:",inline"`
    // name is the name of the group.
    Name string `json:"name" protobuf:"bytes,1,opt,name=name"`
    // versions are the versions supported in this group.
    Versions []GroupVersionForDiscovery `json:"versions" protobuf:"bytes,2,rep,name=versions"`
    //首选版本
    PreferredVersion GroupVersionForDiscovery `json:"preferredVersion,omitempty" protobuf:"bytes,3,opt,name=preferredVersion"`
    ServerAddressByClientCIDRs []ServerAddressByClientCIDR `json:"serverAddressByClientCIDRs,omitempty" protobuf:"bytes,4,rep,name=serverAddressByClientCIDRs"`
}

拥有组名group的资源组,表现形式:<group>/<version>/<resource>,例如:apps/v1/deployments,Http Path以apis为前缀,例如:http://localhost:8080/apis/apps/v1/deployments
没有组名group的资源组被称为核心资源组(Core Groups或Legacy Groups),表表现形式是:/<version>/<resource>,例如:/v1/pod,/v1/configmap,Http Path
以api为前缀,例如:http://localhost:8080/api/v1/pods

资源版本:Version

代码:vendor/k8s.io/apimachinery/pkg/apis/meta/v1/types.go

type APIVersions struct {
    TypeMeta `json:",inline"`
    // versions are the api versions that are available.
    Versions []string `json:"versions" protobuf:"bytes,1,rep,name=versions"`
    ServerAddressByClientCIDRs []ServerAddressByClientCIDR `json:"serverAddressByClientCIDRs" protobuf:"bytes,2,rep,name=serverAddressByClientCIDRs"`
}

语义控制版本:

Alpha,第一阶段,内部测试,v1alpha1、v1alpha2、v2alpha1
Beta ,第二阶段,修复很不完善的地方,仍可能有缺陷,v1beta1、v1beta2、v2beta1
Stable ,第三阶段,达到一定成熟度,可稳定运行,v1,v2,v3

资源版本分为:外部版本和内部版本

外部版本(External Version,例如:Deployment展示:apps/v1)主要用于对外暴露用户请求的接口所使用的对象
内部版本(Internal Version,Deployment展示未:apps/__internal)内部使用,不对外暴露

资源版本和内外部版本有关系吗?

有!有版本号的例如v1属于外部版本(例如:apps/v1/deployments),没有版本号的,拥有runtime.APIVersionInternal(internal)标识的属于内部版本

资源 Resource

一个资源被实例化后被称为资源对象(Resource object,由资源组+资源版本+资源种类组成<group>/<version>,Kind<kind>,例如deployment的资源对象是:apps/v1,Kind=deployment),所有资源对象都是一个实体(Entity),k8s用Entity表示当前状态,可以通过k8s Apiserver查询和更新实体,分为:
持久性实体:Persistent Entity (例如:Deployment,异常退出后会被重新创建)
短暂性实体:Ephemeral Entity也叫Non-Persistent Entity (例如:pod,异常退出后不会重新创建)
代码:vendor/k8s.io/apimachinery/pkg/apis/meta/v1/types.go

// APIResource specifies the name of a resource and whether it is namespaced.
type APIResource struct {
    // 资源名称
    Name string `json:"name" protobuf:"bytes,1,opt,name=name"`
    // 资源单数名称,pods的单数是pod
    SingularName string `json:"singularName" protobuf:"bytes,6,opt,name=singularName"`
    // 是否有命名空间
    Namespaced bool `json:"namespaced" protobuf:"varint,2,opt,name=namespaced"`
    Group string `json:"group,omitempty" protobuf:"bytes,8,opt,name=group"`
    Version string `json:"version,omitempty" protobuf:"bytes,9,opt,name=version"`
    //资源种类
    Kind string `json:"kind" protobuf:"bytes,3,opt,name=kind"`
    //资源操作方法
    Verbs Verbs `json:"verbs" protobuf:"bytes,4,opt,name=verbs"`
    //简称,pod的简称是po
    ShortNames []string `json:"shortNames,omitempty" protobuf:"bytes,5,rep,name=shortNames"`
    //例如:all
    Categories []string `json:"categories,omitempty" protobuf:"bytes,7,rep,name=categories"`
    StorageVersionHash string `json:"storageVersionHash,omitempty" protobuf:"bytes,10,opt,name=storageVersionHash"`
}

资源对象分为:外部版本资源对象和内部版本资源对象

External Object 外部版本资源对象(Versioned Object,拥有资源版本的资源对象),用户通过yaml和json创建资源对象时,所使用的是外部资源对象,外部资源对象通过Alpha、Beta、Stable标识
Internal Object 内部版本资源对象,通过runtime.APIVersionInternal(_internal)标识,不对外暴露,内部使用,主要用于资源版本的转换,例如将v1beta1转为v1版本时候,顺序是先转为内部版本(internal),再转为v1版本,v1beta1→internal→v1(转换函数需要初始化到版注册表schema中)

以pod资源为例: pod外部版本(External Version)

代码:vendor/k8s.io/api/core/v1/types.go (有版本号,有json tags和proto tags,用于序列化和反序列化操作)

type Pod struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
tions.md#spec-and-status
    // +optional
    Spec PodSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`
    Status PodStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
}

pod内部版本(Internal Version)定义:

pkg/apis/core/types.go (没有版本号,也没有json tags)

type Pod struct {
    metav1.TypeMeta
    metav1.ObjectMeta
    Spec PodSpec
    Status PodStatus
}

Resource资源对象的源码定义

kubernetes的资源通过type.go定义在当前资源组/资源版本下所支持的资源类型
代码:pkg/apis/apps/types.go

type Deployment struct {}
type StatefulSet struct {}

内部版本:cd pkg/apis/apps/

tree -L 1
.
├── OWNERS 
├── doc.go
├── fuzzer
├── install
├── register.go
├── types.go
├── v1
├── v1beta1
├── v1beta2
├── validation
└── zz_generated.deepcopy.go
6 directories, 5 files

外部版本:cd pkg/apis/apps/v1

tree -L 1
.
├── conversion.go 
├── conversion_test.go
├── defaults.go
├── defaults_test.go
├── doc.go
├── register.go
├── zz_generated.conversion.go
└── zz_generated.defaults.go
0 directories, 8 files

有没有发现外部版本就在内部版本的里面?
差距是什么?看两个文件:
一个是内部版本的register.go(定义了资源组和内部资源版本),代码:pkg/apis/apps/register.go,有runtime.APIVersionInternal(_internal标识)

// GroupName is the group name use in this package
const GroupName = "apps"

// SchemeGroupVersion is group version used to register these objects
var SchemeGroupVersion = schema.GroupVersion{Group: GroupName, Version: runtime.APIVersionInternal}

外部v1版本registor.go(定义了资源组和v1版本):pkg/apis/apps/v1/register.go(没有runtime标识)

// GroupName is the group name use in this package
const GroupName = "apps"

// SchemeGroupVersion is group version used to register these objects
var SchemeGroupVersion = schema.GroupVersion{Group: GroupName, Version: "v1"}

Resource资源对象注册

每一个资源目录中都有一个资源install/install.go,负责将信息注册到注册表中(schema中),core核心为例:
代码:pkg/apis/core/install/install_test.go

package install

import (
    "k8s.io/apimachinery/pkg/runtime"
    utilruntime "k8s.io/apimachinery/pkg/util/runtime"
    "k8s.io/kubernetes/pkg/api/legacyscheme"
    "k8s.io/kubernetes/pkg/apis/core"
    "k8s.io/kubernetes/pkg/apis/core/v1"
)
func init() {
    Install(legacyscheme.Scheme)
}
// Install registers the API group and adds types to a scheme
func Install(scheme *runtime.Scheme) {
    utilruntime.Must(core.AddToScheme(scheme))
    utilruntime.Must(v1.AddToScheme(scheme))
    utilruntime.Must(scheme.SetVersionPriority(v1.SchemeGroupVersion))
}

legacyscheme.Scheme 全局注册表
core.AddToScheme(scheme)注册核心资源
v1.AddToScheme(scheme)注册core资源组外的外部版本资源
scheme.SetVersionPriority(v1.SchemeGroupVersion) 函数注册资源组的版本顺序,如果有多个资源,排在最前面的为首选版本,这里针对的是没有指定的core核心资源组

Resource首选版本

scheme.SetVersionPriority设置了注册版本顺序是:1,v1beta2,v1beta1
代码:pkg/apis/apps/install/install.go

func Install(scheme *runtime.Scheme) {
    utilruntime.Must(apps.AddToScheme(scheme))
    utilruntime.Must(v1beta1.AddToScheme(scheme))
    utilruntime.Must(v1beta2.AddToScheme(scheme))
    utilruntime.Must(v1.AddToScheme(scheme))
    utilruntime.Must(scheme.SetVersionPriority(v1.SchemeGroupVersion, v1beta2.SchemeGroupVersion, v1beta1.SchemeGroupVersion))
}

当我们使用apps资源组下的deployment资源时候,由于apps资源组下有v1,v1beta2,v1beta1,在一些场景下,如果不指定版本,PreferredVersionAllGroups会优先使用首选版本v1
代码:vendor/k8s.io/apimachinery/pkg/runtime/scheme.go
versionPriority全部是存储对象的外部版本

// PreferredVersionAllGroups returns the most preferred version for every group.
// group ordering is random.
func (s *Scheme) PreferredVersionAllGroups() []schema.GroupVersion {
    ret := []schema.GroupVersion{}
    for group, versions := range s.versionPriority {
        for _, version := range versions {
            ret = append(ret, schema.GroupVersion{Group: group, Version: version})
            break
        }
    }
    for _, observedVersion := range s.observedVersions {
        found := false
        for _, existing := range ret {
            if existing.Group == observedVersion.Group {
                found = true
                break
            }
        }
        if !found {
            ret = append(ret, observedVersion)
        }
    }

    return ret
}

PrioritizedVersionsForGroup 获取指定资源组的资源版本,按照优先顺序返回
PrioritizedVersionsAllGroups 获取所有资源组的资源版本,按照优先顺序返回

Resource支持的两种资源

内置资源(Kubernetes Resource)和自定义资源(Custom Resource)

资源支持的8种操作方法

用这个命令可以打印出k8s支持的所有命令:

kubectl api-resources --no-headers  -o wide | sed 's/.*[//g'| tr -d "]" | tr " " "n" | sort |uniq
create、delete、deletecollection、get、list、patch、update、watch

操作方法可以分为4大类
创建:create
删除:delete,deletecolletcion
更新:update,patch
查询:get,list,watch

deletecollection 是什么?什么时候用到?
举例:删除多个资源的时候用到deletecollection,注意,这里用的的delete不是deletecollection

kubectl delete pods --all
kubectl delete pod -l $key=$value

资源的操作方法通过metav1.Verbs进行描述
代码:vendor/k8s.io/apimachinery/pkg/apis/meta/v1/types.go

type Verbs []string
func (vs Verbs) String() string {
    return fmt.Sprintf("%v", []string(vs))
}

如何查看一个资源对象有哪些操作方法呢?
例如:pod资源对象有create、delete、deletecollection、get、list、patch、update、watch这8种操作方法,pod/logs只有get方法

首先需要知道:每种操作方法由对应一个接口定义的,对应关系如下:
代码:vendor/k8s.io/apiserver/pkg/registry/rest/rest.go
create rest.Creater接口
delete rest.GracefullDeleter接口
deletecollection rest.CollectionDeleter接口
update rest.Updater接口
patch rest.Patcher接口
get rest.Getter接口
list rest.Lister接口
watch rest.Watcher接口

资源对象的操作方法与Storage是相关的,增删查改实际上都是对Storage的操作,对应的资源对象如果实现了以上方法,那么久具有这种操作方法,例如:pod/logs 对应的存储是podrest.LogREST,LogREST实现了rest.Getter接口的,所以pod/logs能被get操作
log对应
podrest.LogREST存储代码:pkg/registry/core/pod/storage/storage.go

// PodStorage includes storage for pods and all sub resources
type PodStorage struct {
    Pod                 *REST
    Binding             *BindingREST
    LegacyBinding       *LegacyBindingREST
    Eviction            *EvictionREST
    Status              *StatusREST
    EphemeralContainers *EphemeralContainersREST
    Log                 *podrest.LogREST
    Proxy               *podrest.ProxyREST
    Exec                *podrest.ExecREST
    Attach              *podrest.AttachREST
    PortForward         *podrest.PortForwardREST
}

// REST implements a RESTStorage for pods
type REST struct {
    *genericregistry.Store
    proxyTransport http.RoundTripper
}

LogREST实现了rest.Getter接口的Get方法,对应代码:pkg/registry/core/pod/rest/log.go

// Get retrieves a runtime.Object that will stream the contents of the pod log
func (r *LogREST) Get(ctx context.Context, name string, opts runtime.Object) (runtime.Object, error) {
    // register the metrics if the context is used.  This assumes sync.Once is fast.  If it's not, it could be an init block.
    registerMetrics()

    logOpts, ok := opts.(*api.PodLogOptions)
    if !ok {
        return nil, fmt.Errorf("invalid options object: %#v", opts)
    }

    countSkipTLSMetric(logOpts.InsecureSkipTLSVerifyBackend)

    if errs := validation.ValidatePodLogOptions(logOpts); len(errs) > 0 {
        return nil, errors.NewInvalid(api.Kind("PodLogOptions"), name, errs)
    }
    location, transport, err := pod.LogLocation(ctx, r.Store, r.KubeletConn, name, logOpts)
    if err != nil {
        return nil, err
    }
    return &genericrest.LocationStreamer{
        Location:                    location,
        Transport:                   transport,
        ContentType:                 "text/plain",
        Flush:                       logOpts.Follow,
        ResponseChecker:             genericrest.NewGenericHttpResponseChecker(api.Resource("pods/log"), name),
        RedirectChecker:             genericrest.PreventRedirects,
        TLSVerificationErrorCounter: podLogsTLSFailure,
    }, nil
}

Pod是REST类型,REST结构体包含genericregistry.Store,Store实现了增删查改各种方法(例如:rest.Getter接口的Get方法,rest.Updater接口的Update方法) ,所以Pod资源对象具有增删查改的各种方法 vendor/k8s.io/apiserver/pkg/registry/generic/registry/store.go

// Get retrieves the item from storage.
func (e *Store) Get(ctx context.Context, name string, options *metav1.GetOptions) (runtime.Object, error) {
    ......
}
func (e *Store) Update(ctx context.Context, name string, objInfo rest.UpdatedObjectInfo, createValidation rest.ValidateObjectFunc, updateValidation rest.ValidateObjectUpdateFunc, forceAllowCreate bool, options *metav1.UpdateOptions) (runtime.Object, bool, error) {
.......
}

资源对象的命名空间

查看所有命名空间

kubectl get namespace
NAME
default  // 默认命名空间,所有未指定的资源放在这里
kube-public // 自动创建的,公共的命名空间,所有用户都能访问 
kube-system // kubernetes系统创建的资源对象都放在这个命名空间
kube-node-lease // 该命名空间包含了每个节点的关联的lease对象,节点租用允许kubelet发送心跳,以便控制节点检测故障

查看哪些资源对象属于命名空间

kubectl api-resources --namespaced=true

查看哪些资源对象不属于命名空间
kubectl api-resources --namespaced=false

资源对象的自定义资源

自定义资源是kubernetes API的扩展,Kubernetes API是kubernetes的前端,是用户集群交互的方式,本质上,API是用来创建、配置、管理kubernetes集群的接口,API可以使用两种方式扩展:

  1. CRD自定义资源,无需编程即可创建,所有自定义资源都可以与k8s内置资源一样使用kubectl或者client-go操作
  2. 通过聚合层实现,需要编程,功能更多

资源对象描述使用Manifest File进行定义,一般定义在yaml或者json中,不举例了
实际状态status和预期的spec定义的状态一致,这是k8s管理资源对象要做的事情

列出k8s集群中支持的资源组和资源版本,表现形式为<group>/<version>
kubectl api-versions

列出当前kuberneteus版本支持的resources列表
kubectl api-resources
kubectl api-resources --help
kubectl api-resources -o wide

获取特定api组的api资源
kubectl api-resources --api-group=apps

资源对象的runtime.Object类型接口

&core.Pod()和app.Deployment()都底层都是runtime.Object类型接口
代码:vendor/k8s.io/apimachinery/pkg/runtime/interfaces.go

type Object interface {
    GetObjectKind() schema.ObjectKind
    DeepCopyObject() Object
}

schema.ObjectKind接口包含SetGroupVersionKind和GroupVersionKind两个方法
代码:

type ObjectKind interface {
    // SetGroupVersionKind sets or clears the intended serialized kind of an object. Passing kind nil
    // should clear the current setting.
    SetGroupVersionKind(kind GroupVersionKind)
    // GroupVersionKind returns the stored group, version, and kind of an object, or an empty struct
    // if the object does not expose or provide these fields.
    GroupVersionKind() GroupVersionKind
}
// EmptyObjectKind implements the ObjectKind interface as a noop
var EmptyObjectKind = emptyObjectKind{}

type emptyObjectKind struct{}

// SetGroupVersionKind implements the ObjectKind interface
func (emptyObjectKind) SetGroupVersionKind(gvk GroupVersionKind) {}

// GroupVersionKind implements the ObjectKind interface
func (emptyObjectKind) GroupVersionKind() GroupVersionKind { return GroupVersionKind{} }

那么实现SetGroupVersionKind和GroupVersionKind两个方法,就实现了schema.ObjectKind接口,那core.Pod结构体是否实现了这两个方法呢?我们看下pod代码:
pod定义:pkg/apis/core/types.go,包含了metav1.TypeMeta

type Pod struct {
    metav1.TypeMeta
    // +optional
    metav1.ObjectMeta

    // Spec defines the behavior of a pod.
    // +optional
    Spec PodSpec

    // Status represents the current information about a pod. This data may not be up
    // to date.
    // +optional
    Status PodStatus
}

再看代码:staging/src/k8s.io/apimachinery/pkg/runtime/register.go
*TypeMeta实现了SetGroupVersionKind方法和GroupVersionKind方法,所以pod实现了schema.ObjectKind接口,又因为kubernetes每个资源对象都实现了DeepCopyObject方法(vendor/k8s.io/apimachinery/pkg/apis/meta/v1/zz_generated.deepcopy.go)所以core.Pod()结构实现了runtime.Object类型接口

package runtime
import "k8s.io/apimachinery/pkg/runtime/schema"
// SetGroupVersionKind satisfies the ObjectKind interface for all objects that embed TypeMeta
func (obj *TypeMeta) SetGroupVersionKind(gvk schema.GroupVersionKind) {
    obj.APIVersion, obj.Kind = gvk.ToAPIVersionAndKind()
}
// GroupVersionKind satisfies the ObjectKind interface for all objects that embed TypeMeta
func (obj *TypeMeta) GroupVersionKind() schema.GroupVersionKind {
    return schema.FromAPIVersionAndKind(obj.APIVersion, obj.Kind)
}
func (obj *TypeMeta) GetObjectKind() schema.ObjectKind { return obj }

资源对象结构化数据和非结构化数据

结构化数据是类型确定的数据结构,例如json数据,如果要使用这种数据,创建一个struct结构体,通过go语言的json库反序列化操作,转为struct结构体;
非结构化的数据是类型不确定的数据结构,例如:description的类型不清楚,可能是int,也可能是string,这种可以通过类型断言方式去判断

{
"id": 1,
"name": "abc",
"description": ... 
}
if description, ok := result["description"],{string}; ok { fmt.Println(description)
}

非结构化数据代码定义,Object map[string]interface{} 值是接口类型
vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured/unstructured.go

type Unstructured struct {
    // Object is a JSON compatible map with string, float, int, bool, []interface{}, or
    // map[string]interface{}
    // children.
    Object map[string]interface{}
}

vendor/k8s.io/apimachinery/pkg/runtime/interfaces.go

// Unstructured objects store values as map[string]interface{}, with only values that can be serialized
// to JSON allowed.
type Unstructured interface {
    Object
    // NewEmptyInstance returns a new instance of the concrete type containing only kind/apiVersion and no other data.
    // This should be called instead of reflect.New() for unstructured types because the go type alone does not preserve kind/apiVersion info.
    NewEmptyInstance() Unstructured
    // UnstructuredContent returns a non-nil map with this object's contents. Values may be
    // []interface{}, map[string]interface{}, or any primitive type. Contents are typically serialized to
    // and from JSON. SetUnstructuredContent should be used to mutate the contents.
    UnstructuredContent() map[string]interface{}
    // SetUnstructuredContent updates the object content to match the provided map.
    SetUnstructuredContent(map[string]interface{})
    // IsList returns true if this type is a list or matches the list convention - has an array called "items".
    IsList() bool
    // EachListItem should pass a single item out of the list as an Object to the provided function. Any
    // error should terminate the iteration. If IsList() returns false, this method should return an error
    // instead of calling the provided function.
    EachListItem(func(Object) error) error
}

资源列表:ResourceList

vendor/k8s.io/apimachinery/pkg/apis/meta/v1/types.go

// APIResourceList is a list of APIResource, it is used to expose the name of the
// resources supported in a specific group and version, and if the resource
// is namespaced.
type APIResourceList struct {
    TypeMeta `json:",inline"`
    // groupVersion is the group and version this APIResourceList is for.
    GroupVersion string `json:"groupVersion" protobuf:"bytes,1,opt,name=groupVersion"`
    // resources contains the name of the resources and if they are namespaced.
    APIResources []APIResource `json:"resources" protobuf:"bytes,2,rep,name=resources"`
}

看一个例子:代码:pkg/controller/namespace/deletion/namespaced_resources_deleter_test.go

// testResources returns a mocked up set of resources across different api groups for testing namespace controller.
func testResources() []*metav1.APIResourceList {
    results := []*metav1.APIResourceList{
        {
            GroupVersion: "v1",
            APIResources: []metav1.APIResource{
                {
                    Name:       "pods",
                    Namespaced: true,
                    Kind:       "Pod",
                    Verbs:      []string{"get", "list", "delete", "deletecollection", "create", "update"},
                },
                {
                    Name:       "services",
                    Namespaced: true,
                    Kind:       "Service",
                    Verbs:      []string{"get", "list", "delete", "deletecollection", "create", "update"},
                },
            },
        },
        {
            GroupVersion: "apps/v1",
            APIResources: []metav1.APIResource{
                {
                    Name:       "deployments",
                    Namespaced: true,
                    Kind:       "Deployment",
                    Verbs:      []string{"get", "list", "delete", "deletecollection", "create", "update"},
                },
            },
        },
    }
    return results
}

资源的数据结构

GVR:Group Version Resource 资源组名称或者资源版本及资源名称
GV:Group Version 资源组以及资源版本
GVS:Group Versions 资源组内多个资源版本
GVK:Group VersionKind 资源组或者资源版本以及资源种类
GK:Group Kind 资源组或资源种类
GR:Group Resource 资源组或者资源

源码:k8s.io/apimachinery/pkg/runtime/schema/group_version.go

type GroupVersionResource struct {
    Group    string
    Version  string
    Resource string
}

//以deployment以例:
type GroupVersionResource{
    Group:"apps",
    Version:"v1",
    Resource:"Deployment"
}

Kubernetes API

kubernetes是一个REST api驱动系统,开启服务器的访问权限,最简单的方法是使用kube-proxy,这是一个反向代理自动定位API服务器,然后允许用户与默认访问进行交互

//一窗口执行:
kubectl proxy --port 8080
//另外一个窗口访问:
curl --request GET --url http://localhost:8080/api/v1
作者:golang中国
golang中国

本文》有 3828 条评论

留下一个回复