`<< Back <../../kubernetes>`__ .. _4-component-level-architecture: 4. Component Level Architecture =============================== .. raw:: html

scope

.. _table-of-contents-: Table of Contents ----------------- - `4. Component Level Architecture <#4-component-level-architecture>`__ - `4.1 Introduction <#41-introduction>`__ - `4.2 Kubernetes Node <#42-kubernetes-node>`__ - `4.3 Kubernetes <#43-kubernetes>`__ - `4.4 Container runtimes <#44-container-runtimes>`__ - `4.5 Networking solutions <#45-networking-solutions>`__ - `4.6 Storage components <#46-storage-components>`__ - `4.7 Service meshes <#47-service-meshes>`__ - `4.8 Kubernetes Application package manager <#48-kubernetes-application-package-manager>`__ - `4.9 Kubernetes workloads <#49-kubernetes-workloads>`__ - `4.10 Additional required components <#410-additional-required-components>`__ .. _41-introduction: 4.1 Introduction ---------------- This chapter describes in detail the Kubernetes Reference Architecture in terms of the functional capabilities and how they relate to the Reference Model requirements, i.e. how the infrastructure profiles are determined, documented and delivered. The specifications defined in this chapter will be detailed with unique identifiers, which will follow the pattern: ``ra2.
.``, e.g. ``ra2.ch.001`` for the first requirement in the Kubernetes Node section. These specifications will then be used as requirements input for the Kubernetes Reference Implementation and any vendor or community implementations. Figure 4-1 below shows the architectural components that are described in the subsequent sections of this chapter. .. raw:: html

Kubernetes Reference Architecture

Figure 4-1: Kubernetes Reference Architecture

.. _42-kubernetes-node: 4.2 Kubernetes Node ------------------- This section describes the configuration that will be applied to the physical or virtual machine and an installed Operating System. In order for a Kubernetes Node to be conformant with the Reference Architecture it must be implemented as per the following specifications: +-------------+-------------+-------------+-------------+-------------+ | Ref | Sp | Details | Requirement | Reference | | | ecification | | Trace | Imp | | | | | | lementation | | | | | | Trace | +=============+=============+=============+=============+=============+ | ``r | Huge Pages | When | `infr | `4 | | a2.ch.001`` | | hosting | a.com.cfg.0 | .3.1 `__ | etal-infrat | | | | enable Huge | | ructure>`__ | | | | Pages | | | | | | (2048KiB | | | | | | and | | | | | | 1048576KiB) | | | | | | within the | | | | | | Kubernetes | | | | | | Node OS, | | | | | | exposing | | | | | | schedulable | | | | | | resources | | | | | | ``huge | | | | | | pages-2Mi`` | | | | | | and | | | | | | ``hugep | | | | | | ages-1Gi``. | | | +-------------+-------------+-------------+-------------+-------------+ | ``r | SR-IOV | When | `e.cap.0 | `3.3 `__ | rements>`__ | | | | physical | | | | | | machines on | | | | | | which the | | | | | | Kubernetes | | | | | | Nodes run | | | | | | must be | | | | | | equipped | | | | | | with NICs | | | | | | that are | | | | | | SR-IOV | | | | | | capable. | | | +-------------+-------------+-------------+-------------+-------------+ | ``r | SR-IOV | When | `e.cap.0 | `4 | | a2.ch.003`` | Virtual | hosting | 13 <./chapt | .3.1 `__ | n-on-bare-m | | | | virtual | | etal-infrat | | | | functions | | ructure>`__ | | | | (VFs) must | | | | | | be | | | | | | configured | | | | | | within the | | | | | | Kubernetes | | | | | | Node OS, as | | | | | | the SR-IOV | | | | | | Device | | | | | | Plugin does | | | | | | not manage | | | | | | the | | | | | | creation of | | | | | | these VFs. | | | +-------------+-------------+-------------+-------------+-------------+ | ``r | CPU | SMT must be | `infra.h | `3.3 `__ | | | | Node runs. | rements>`__ | | +-------------+-------------+-------------+-------------+-------------+ | ``r | CPU | For | `infr | | | a2.ch.005`` | Allocation | Kubernetes | a.com.cfg.0 | | | | Ratio - VMs | nodes | 01 <./chapt | | | | | running as | er02.md#223 | | | | | Virtual | -cloud-infr | | | | | Machines, | astructure- | | | | | ensure the | software-pr | | | | | CPU | ofile-requi | | | | | allocation | rements>`__ | | | | | ratio | | | | | | between | | | | | | vCPU and | | | | | | physical | | | | | | CPU core is | | | | | | 1:1. | | | +-------------+-------------+-------------+-------------+-------------+ | ``r | CPU | To ensure | `infr | `3.3 `__ | | | | 1:1, the | rements>`__ | | | | | sum of CPU | | | | | | requests | | | | | | and limits | | | | | | by | | | | | | containers | | | | | | in Pod | | | | | | spe | | | | | | cifications | | | | | | must remain | | | | | | less than | | | | | | the | | | | | | allocatable | | | | | | quantity of | | | | | | CPU | | | | | | resources | | | | | | (i.e. | | | | | | ` | | | | | | `requests.c | | | | | | pu < alloca | | | | | | table.cpu`` | | | | | | and | | | | | | ` | | | | | | `limits.cpu | | | | | | < allocata | | | | | | ble.cpu``). | | | +-------------+-------------+-------------+-------------+-------------+ | ``r | IP | To support | `req.inf. | | | a2.ch.007`` | v6DualStack | IPv4/IPv6 | ntw.04 <./c | | | | | dual stack | hapter02.md | | | | | networking, | #23-kuberne | | | | | the | tes-archite | | | | | Kubernetes | cture-requi | | | | | Node OS | rements>`__ | | | | | must | | | | | | support and | | | | | | be | | | | | | allocated | | | | | | routable | | | | | | IPv4 and | | | | | | IPv6 | | | | | | addresses. | | | +-------------+-------------+-------------+-------------+-------------+ | ``r | Physical | The | `infra.hw | `3.3 `__ | | | | with at | ements>`__\ | | | | | least 2 | \ `infra.h | | | | | physical | w.cpu.cfg.0 | | | | | sockets, | 02 <./chapt | | | | | each of at | er02.md#224 | | | | | least 20 | -cloud-infr | | | | | CPU cores. | astructure- | | | | | | hardware-pr | | | | | | ofile-requi | | | | | | rements>`__ | | +-------------+-------------+-------------+-------------+-------------+ | ``r | Physical | The | ` | `3.3 `__ | | | | with Sold | ofile-requi | | | | | State | rements>`__ | | | | | Drives | | | | | | (SSDs). | | | +-------------+-------------+-------------+-------------+-------------+ | ``r | Local | The | `e.cap.0 | `3.3 `__ | rements>`__ | | | | 320GB for | | | | | | unpacking | | | | | | and | | | | | | executing | | | | | | containers. | | | | | | Note, extra | | | | | | should be | | | | | | provisioned | | | | | | to cater | | | | | | for any | | | | | | overhead | | | | | | required by | | | | | | the | | | | | | Operating | | | | | | System and | | | | | | any | | | | | | required OS | | | | | | processes | | | | | | such as the | | | | | | container | | | | | | runtime, | | | | | | Kubernetes | | | | | | agents, | | | | | | etc. | | | +-------------+-------------+-------------+-------------+-------------+ | ``r | Virtual | If using | `e.cap.0 | | | a2.ch.011`` | Node CPU | VMs, the | 01 <./chapt | | | | Quantity | Kubernetes | er02.md#221 | | | | | Nodes must | -cloud-infr | | | | | be equipped | astructure- | | | | | with at | software-pr | | | | | least 16 | ofile-capab | | | | | vCPUs. | ilities>`__ | | | | | Note, extra | | | | | | should be | | | | | | provisioned | | | | | | to cater | | | | | | for any | | | | | | overhead | | | | | | required by | | | | | | the | | | | | | Operating | | | | | | System and | | | | | | any | | | | | | required OS | | | | | | processes | | | | | | such as the | | | | | | container | | | | | | runtime, | | | | | | Kubernetes | | | | | | agents, | | | | | | etc. | | | +-------------+-------------+-------------+-------------+-------------+ | ``r | Kubernetes | The | `e.cap.0 | `3.3 `__ | rements>`__ | | | | should be | | | | | | provisioned | | | | | | to cater | | | | | | for any | | | | | | overhead | | | | | | required by | | | | | | the | | | | | | Operating | | | | | | System and | | | | | | any | | | | | | required OS | | | | | | processes | | | | | | such as the | | | | | | container | | | | | | runtime, | | | | | | Kubernetes | | | | | | agents, | | | | | | etc. | | | +-------------+-------------+-------------+-------------+-------------+ | ``r | Physical | The | `infra.h | `3.3 `__ | | | | with at | rements>`__ | | | | | least four | | | | | | (4) Network | | | | | | Interface | | | | | | Card (NIC) | | | | | | ports. | | | +-------------+-------------+-------------+-------------+-------------+ | ``r | Physical | The NIC | `infra.h | `3.3 `__ | | | | Nodes run | rements>`__ | | | | | for | | | | | | workloads | | | | | | matching | | | | | | the Basic | | | | | | Profile | | | | | | must be at | | | | | | least | | | | | | 10Gbps. | | | +-------------+-------------+-------------+-------------+-------------+ | ``r | Physical | The NIC | `infra.h | `3.3 `__ | | | | Nodes run | rements>`__ | | | | | for | | | | | | workloads | | | | | | matching | | | | | | the Network | | | | | | Intensive | | | | | | profile | | | | | | must be at | | | | | | least | | | | | | 25Gbps. | | | +-------------+-------------+-------------+-------------+-------------+ | ``r | Physical | The | | | | a2.ch.016`` | PCIe slots | physical | | | | | | machines on | | | | | | which the | | | | | | Kubernetes | | | | | | Nodes run | | | | | | must be | | | | | | equipped | | | | | | with at | | | | | | least eight | | | | | | (8) Gen3.0 | | | | | | PCIe slots, | | | | | | each with | | | | | | at least | | | | | | eight (8) | | | | | | lanes. | | | +-------------+-------------+-------------+-------------+-------------+ | ``r | Immutable | Whether | `req.gen. | `4 | | a2.ch.017`` | inf | physical or | cnt.02 <./c | .3.1 `__ | installatio | | | | Node is not | | n-on-bare-m | | | | changed | | etal-infrat | | | | after it is | | ructure>`__ | | | | made ready | | | | | | for use. | | | | | | New changes | | | | | | to the | | | | | | Kubernetes | | | | | | Node are | | | | | | rolled out | | | | | | as new | | | | | | instances. | | | | | | This covers | | | | | | any changes | | | | | | from BIOS | | | | | | through | | | | | | Operating | | | | | | System to | | | | | | running | | | | | | processes | | | | | | and all | | | | | | associated | | | | | | conf | | | | | | igurations. | | | +-------------+-------------+-------------+-------------+-------------+ | ``r | NFD | `Node | TBD | `4 | | a2.ch.018`` | | Feature | | .3.1 `__ | | | | started/ind | | | | | | ex.html>`__ | | | | | | must be | | | | | | used to | | | | | | advertise | | | | | | the | | | | | | detailed | | | | | | software | | | | | | and | | | | | | hardware | | | | | | c | | | | | | apabilities | | | | | | of each | | | | | | node in the | | | | | | Kubernetes | | | | | | Cluster. | | | +-------------+-------------+-------------+-------------+-------------+ .. raw:: html

Table 4-1: Node Specifications

.. _43-kubernetes: 4.3 Kubernetes -------------- In order for the Kubernetes components to be conformant with the Reference Architecture they must be implemented as per the following specifications: +-------------+-------------+-------------+-------------+-------------+ | Ref | Sp | Details | Requirement | Reference | | | ecification | | Trace | Imp | | | | | | lementation | | | | | | Trace | +=============+=============+=============+=============+=============+ | ``ra | Kubernetes | The | `req.gen. | `4 | | 2.k8s.001`` | Conformance | Kubernetes | cnt.03 <./c | .3.1 `__ | installatio | | | | imp | | n-on-bare-m | | | | lementation | | etal-infrat | | | | **must** be | | ructure>`__ | | | | listed in | | | | | | the | | | | | | `Kubernetes | | | | | | Di | | | | | | stributions | | | | | | and | | | | | | Platforms | | | | | | document | | | | | | `__ | | | | | | and marked | | | | | | (X) as | | | | | | conformant | | | | | | for the | | | | | | Kubernetes | | | | | | version | | | | | | defined in | | | | | | `READM | | | | | | E <../READM | | | | | | E.md#requir | | | | | | ed-versions | | | | | | -of-most-im | | | | | | portant-com | | | | | | ponents>`__ | | | | | | under | | | | | | "Required | | | | | | versions of | | | | | | most | | | | | | important | | | | | | c | | | | | | omponents". | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Highly | An | `req | `4 | | 2.k8s.002`` | available | imp | .gen.rsl.02 | .3.1 `__ | etal-infrat | | | | running the | | ructure>`__ | | | | etcd | | | | | | service | | | | | | (can be | | | | | | colocated | | | | | | on the | | | | | | master | | | | | | nodes, or | | | | | | can run on | | | | | | separate | | | | | | nodes, but | | | | | | not on | | | | | | worker | | | | | | nodes). | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Highly | An | `req.gen.rs | | | 2.k8s.003`` | available | imp | l.02 <./cha | | | | control | lementation | pter02.md#2 | | | | plane | must | 3-kubernete | | | | | consist of | s-architect | | | | | at least | ure-require | | | | | one master | ments>`__\ | | | | | node per | \ `req.gen. | | | | | a | avl.01 <./c | | | | | vailability | hapter02.md | | | | | zone or | #23-kuberne | | | | | fault | tes-archite | | | | | domain to | cture-requi | | | | | ensure the | rements>`__ | | | | | high | | | | | | a | | | | | | vailability | | | | | | and | | | | | | resilience | | | | | | of the | | | | | | Kubernetes | | | | | | control | | | | | | plane | | | | | | services. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Control | A master | `req.gen.rs | `4 | | 2.k8s.012`` | plane | node must | l.02 <./cha | .3.1 `__\ | installatio | | | | plane | \ `req.gen. | n-on-bare-m | | | | services: | avl.01 <./c | etal-infrat | | | | ``kube-a | hapter02.md | ructure>`__ | | | | piserver``, | #23-kuberne | | | | | ``kube- | tes-archite | | | | | scheduler`` | cture-requi | | | | | and | rements>`__ | | | | | ``kube | | | | | | -controller | | | | | | -manager``. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Highly | An | `req | | | 2.k8s.004`` | available | imp | .gen.rsl.01 | | | | worker | lementation | <./chapter | | | | nodes | must | 02.md#23-ku | | | | | consist of | bernetes-ar | | | | | at least | chitecture- | | | | | one worker | requirement | | | | | node per | s>`__\ \ `r | | | | | a | eq.gen.avl. | | | | | vailability | 01 <./chapt | | | | | zone or | er02.md#23- | | | | | fault | kubernetes- | | | | | domain to | architectur | | | | | ensure the | e-requireme | | | | | high | nts>`__\ \ | | | | | a | `req.kcm.ge | | | | | vailability | n.02 <./cha | | | | | and | pter02.md#2 | | | | | resilience | 3-kubernete | | | | | of | s-architect | | | | | workloads | ure-require | | | | | managed by | ments>`__\ | | | | | Kubernetes | \ `req.inf. | | | | | | com.01 <./c | | | | | | hapter02.md | | | | | | #23-kuberne | | | | | | tes-archite | | | | | | cture-requi | | | | | | rements>`__ | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Kubernetes | In | TBC | | | 2.k8s.005`` | API Version | alignment | | | | | | with the | | | | | | `Kubernetes | | | | | | version | | | | | | support | | | | | | polic | | | | | | y `__, | | | | | | an | | | | | | imp | | | | | | lementation | | | | | | **must** | | | | | | use | | | | | | Kubernetes | | | | | | version as | | | | | | per the | | | | | | s | | | | | | ubcomponent | | | | | | versions | | | | | | table in | | | | | | `READM | | | | | | E <../READM | | | | | | E.md#requir | | | | | | ed-versions | | | | | | -of-most-im | | | | | | portant-com | | | | | | ponents>`__ | | | | | | under | | | | | | "Required | | | | | | versions of | | | | | | most | | | | | | important | | | | | | c | | | | | | omponents". | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | NUMA | When | `e.cap | | | 2.k8s.006`` | Support | hosting | .007 `__ | | | | | ``Topolo | `infr | | | | | gyManager`` | a.com.cfg.0 | | | | | and | 02 <./chapt | | | | | ``C | er02.md#223 | | | | | PUManager`` | -cloud-infr | | | | | feature | astructure- | | | | | gates must | software-pr | | | | | be enabled | ofile-requi | | | | | and | rements>`__ | | | | | configured | `infra.h | | | | | on the | w.cpu.cfg.0 | | | | | kubelet | 03 <./chapt | | | | | (note, | er02.md#224 | | | | | Topo | -cloud-infr | | | | | logyManager | astructure- | | | | | is enabled | hardware-pr | | | | | by default | ofile-requi | | | | | in | rements>`__ | | | | | Kubernetes | | | | | | v1.18 and | | | | | | later, with | | | | | | CPUManager | | | | | | enabled by | | | | | | default in | | | | | | Kubernetes | | | | | | v1.10 and | | | | | | later). | | | | | | ` | | | | | | `--feature- | | | | | | gates="..., | | | | | | TopologyMan | | | | | | ager=true,C | | | | | | PUManager=t | | | | | | rue" --topo | | | | | | logy-manage | | | | | | r-policy=si | | | | | | ngle-numa-n | | | | | | ode --cpu-m | | | | | | anager-poli | | | | | | cy=static`` | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | De | When | Various, | `4 | | 2.k8s.007`` | vicePlugins | hosting | e.g. | .3.1 `__ | ructure>`__ | | | | feature | | | | | | gate must | | | | | | be enabled | | | | | | (note, this | | | | | | is enabled | | | | | | by default | | | | | | in | | | | | | Kubernetes | | | | | | v1.10 or | | | | | | later). | | | | | | ``-- | | | | | | feature-gat | | | | | | es="...,Dev | | | | | | icePlugins= | | | | | | true,..."`` | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | System | To avoid | `i.cap | | | 2.k8s.008`` | Resource | resource | .014 `__ | | | | | a | | | | | | rchitecture | | | | | | **must** | | | | | | reserve | | | | | | compute | | | | | | resources | | | | | | for system | | | | | | daemons and | | | | | | Kubernetes | | | | | | system | | | | | | daemons | | | | | | such as | | | | | | kubelet, | | | | | | container | | | | | | runtime, | | | | | | etc. Use | | | | | | the | | | | | | following | | | | | | kubelet | | | | | | flags: | | | | | | ``-- | | | | | | reserved-cp | | | | | | us=[a-z]``, | | | | | | using two | | | | | | of ``a-z`` | | | | | | to reserve | | | | | | 2 SMT | | | | | | threads. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | CPU Pinning | When | `infr | | | 2.k8s.009`` | | hosting | a.com.cfg.0 | | | | | workloads | 03 <./chapt | | | | | matching | er02.md#223 | | | | | the Network | -cloud-infr | | | | | Intensive | astructure- | | | | | profile, in | software-pr | | | | | order to | ofile-requi | | | | | support CPU | rements>`__ | | | | | Pinning, | | | | | | the kubelet | | | | | | must be | | | | | | started | | | | | | with the | | | | | | ``--cpu-m | | | | | | anager-poli | | | | | | cy=static`` | | | | | | option. | | | | | | (Note, only | | | | | | containers | | | | | | in | | | | | | ``G | | | | | | uaranteed`` | | | | | | pods - | | | | | | where CPU | | | | | | resource | | | | | | ` | | | | | | `requests`` | | | | | | and | | | | | | ``limits`` | | | | | | are | | | | | | identical - | | | | | | and | | | | | | configured | | | | | | with | | | | | | posit | | | | | | ive-integer | | | | | | CPU | | | | | | ` | | | | | | `requests`` | | | | | | will take | | | | | | advantage | | | | | | of this. | | | | | | All other | | | | | | Pods will | | | | | | run on CPUs | | | | | | in the | | | | | | remaining | | | | | | shared | | | | | | pool.) | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | IP | To support | `req.inf. | | | 2.k8s.010`` | v6DualStack | IPv6 and | ntw.04 <./c | | | | | IPv4, the | hapter02.md | | | | | ``IPv6 | #23-kuberne | | | | | DualStack`` | tes-archite | | | | | feature | cture-requi | | | | | gate must | rements>`__ | | | | | be enabled | | | | | | on various | | | | | | components | | | | | | (requires | | | | | | Kubernetes | | | | | | v1.16 or | | | | | | later). | | | | | | kube | | | | | | -apiserver: | | | | | | ``--feat | | | | | | ure-gates=" | | | | | | IPv6DualSta | | | | | | ck=true"``. | | | | | | ku | | | | | | be-controll | | | | | | er-manager: | | | | | | ``--featur | | | | | | e-gates="IP | | | | | | v6DualStack | | | | | | =true" --cl | | | | | | uster-cidr= | | | | | | | | | | | | , --service | | | | | | -cluster-ip | | | | | | -range=, -- | | | | | | node-cidr-m | | | | | | ask-size-ip | | | | | | v4 ¦ --node | | | | | | -cidr-mask- | | | | | | size-ipv6`` | | | | | | defaults to | | | | | | /24 for | | | | | | IPv4 and | | | | | | /64 for | | | | | | IPv6. | | | | | | kubelet: | | | | | | ``--feat | | | | | | ure-gates=" | | | | | | IPv6DualSta | | | | | | ck=true"``. | | | | | | kube-proxy: | | | | | | `` | | | | | | --cluster-c | | | | | | idr=, --fea | | | | | | ture-gates= | | | | | | "IPv6DualSt | | | | | | ack=true"`` | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Anuket | To clearly | | | | 2.k8s.011`` | profile | identify | | | | | labels | which | | | | | | worker | | | | | | nodes are | | | | | | compliant | | | | | | with the | | | | | | different | | | | | | profiles | | | | | | defined by | | | | | | Anuket the | | | | | | worker | | | | | | nodes must | | | | | | be labelled | | | | | | according | | | | | | to the | | | | | | following | | | | | | pattern: an | | | | | | ``anu | | | | | | ket.io/prof | | | | | | ile/basic`` | | | | | | label must | | | | | | be set to | | | | | | ``true`` on | | | | | | the worker | | | | | | node if it | | | | | | can fulfil | | | | | | the | | | | | | r | | | | | | equirements | | | | | | of the | | | | | | basic | | | | | | profile and | | | | | | an | | | | | | ``anuk | | | | | | et.io/profi | | | | | | le/network- | | | | | | intensive`` | | | | | | label must | | | | | | be set to | | | | | | ``true`` on | | | | | | the worker | | | | | | node if it | | | | | | can fulfil | | | | | | the | | | | | | r | | | | | | equirements | | | | | | of the | | | | | | network | | | | | | intensive | | | | | | profile. | | | | | | The | | | | | | r | | | | | | equirements | | | | | | for both | | | | | | profiles | | | | | | can be | | | | | | found in | | | | | | `chapter | | | | | | 2 <./chap | | | | | | ter02.md#22 | | | | | | -reference- | | | | | | model-requi | | | | | | rements>`__ | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Kubernetes | Kubernetes | ` | | | 2.k8s.012`` | APIs | `Alpha | req.int.api | | | | | API < | .03 <./chap | | | | | https://kub | ter02.md#22 | | | | | ernetes.io/ | -reference- | | | | | docs/refere | model-requi | | | | | nce/using-a | rements>`__ | | | | | pi/#api-ver | | | | | | sioning>`__ | | | | | | are | | | | | | recommended | | | | | | only for | | | | | | testing, | | | | | | therefore | | | | | | all Alpha | | | | | | APIs | | | | | | **must** be | | | | | | disabled. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Kubernetes | Backward | ` | | | 2.k8s.013`` | APIs | co | req.int.api | | | | | mpatibility | .04 <./chap | | | | | of all | ter02.md#22 | | | | | supported | -reference- | | | | | GA and Beta | model-requi | | | | | APIs of | rements>`__ | | | | | Kubernetes | | | | | | **must** be | | | | | | supported. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Security | Kubernetes | `infra.net | | | 2.k8s.014`` | Groups | **must** | .cfg.004 `__ | | +-------------+-------------+-------------+-------------+-------------+ .. raw:: html

Table 4-2: Kubernetes Specifications

.. _44-container-runtimes: 4.4 Container runtimes ---------------------- +-------------+-------------+-------------+-------------+-------------+ | Ref | Sp | Details | Requirement | Reference | | | ecification | | Trace | Imp | | | | | | lementation | | | | | | Trace | +=============+=============+=============+=============+=============+ | ``ra | Conformance | The | `req.ge | `4 | | 2.crt.001`` | with OCI | container | n.ost.01 `__ | installatio | | | | 1.0 `__ | | | | tainers/run | | | | | | time-spec/b | | | | | | lob/master/ | | | | | | spec.md>`__ | | | | | | (Open | | | | | | Container | | | | | | Initiative | | | | | | 1.0) | | | | | | spe | | | | | | cification. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Kubernetes | The | `req.ge | `4 | | 2.crt.002`` | Container | Kubernetes | n.ost.01 `__ | installatio | | | | `Kubernetes | | n-on-bare-m | | | | Container | | etal-infrat | | | | Runtime | | ructure>`__ | | | | Interface | | | | | | (CRI | | | | | | ) `__ | | | +-------------+-------------+-------------+-------------+-------------+ .. raw:: html

Table 4-3: Container Runtime Specifications

.. _45-networking-solutions: 4.5 Networking solutions ------------------------ In order for the networking solution(s) to be conformant with the Reference Architecture they must be implemented as per the following specifications: +-------------+-------------+-------------+-------------+-------------+ | Ref | Sp | Details | Requirement | Reference | | | ecification | | Trace | Imp | | | | | | lementation | | | | | | Trace | +=============+=============+=============+=============+=============+ | ``ra | Centralised | The | `req.in | `4 | | 2.ntw.001`` | network | networking | f.ntw.03 `__ | installatio | | | | must be | | n-on-bare-m | | | | a | | etal-infrat | | | | dministered | | ructure>`__ | | | | through the | | | | | | Kubernetes | | | | | | API using | | | | | | native | | | | | | Kubernetes | | | | | | API | | | | | | resources | | | | | | and | | | | | | objects, or | | | | | | Custom | | | | | | Resources. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Default Pod | The | `req.ge | `4 | | 2.ntw.002`` | Network - | networking | n.ost.01 `__ | installatio | | | | must use a | \ \ `req.in | n-on-bare-m | | | | CNI | f.ntw.08 `__ | | | | Network | #23-kuberne | | | | | Plugin for | tes-archite | | | | | the Default | cture-requi | | | | | Pod | rements>`__ | | | | | Network, as | | | | | | the | | | | | | alternative | | | | | | (kubenet) | | | | | | does not | | | | | | support | | | | | | cross-node | | | | | | networking | | | | | | or Network | | | | | | Policies. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Multiple | The | `e.cap | `4 | | 2.ntw.003`` | connection | networking | .004 `__ | n-on-bare-m | | | | support the | | etal-infrat | | | | capability | | ructure>`__ | | | | to connect | | | | | | at least | | | | | | FIVE | | | | | | connection | | | | | | points to | | | | | | each Pod, | | | | | | which are | | | | | | additional | | | | | | to the | | | | | | default | | | | | | connection | | | | | | point | | | | | | managed by | | | | | | the default | | | | | | Pod network | | | | | | CNI plugin. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Multiple | The | `req.in | `4 | | 2.ntw.004`` | connection | networking | f.ntw.03 `__ | installatio | | | | must ensure | | n-on-bare-m | | | | that all | | etal-infrat | | | | additional | | ructure>`__ | | | | non-default | | | | | | connection | | | | | | points are | | | | | | requested | | | | | | by Pods | | | | | | using | | | | | | standard | | | | | | Kubernetes | | | | | | resource | | | | | | scheduling | | | | | | mechanisms | | | | | | such as | | | | | | annotations | | | | | | or | | | | | | container | | | | | | resource | | | | | | requests | | | | | | and limits. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | M | The | `req.in | `4 | | 2.ntw.005`` | ultiplexer/ | networking | f.ntw.06 `__ | installatio | | | | may use a | \ \ `req.in | n-on-bare-m | | | | mu | f.ntw.07 `__ | | | | eta-plugin. | #23-kuberne | | | | | | tes-archite | | | | | | cture-requi | | | | | | rements>`__ | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | M | If used, | `req.ge | `4 | | 2.ntw.006`` | ultiplexer/ | the | n.ost.01 `__ | installatio | | | | integrate | | n-on-bare-m | | | | with the | | etal-infrat | | | | Kubernetes | | ructure>`__ | | | | control | | | | | | plane via | | | | | | CNI. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | M | If used, | `req.ge | `4 | | 2.ntw.007`` | ultiplexer/ | the | n.ost.01 `__ | installatio | | | | support the | \ \ `req.in | n-on-bare-m | | | | use of | f.ntw.06 `__ | | | | CNI | #23-kuberne | | | | | -conformant | tes-archite | | | | | Network | cture-requi | | | | | Plugins. | rements>`__ | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | SR-IOV | When | `e.cap | `4 | | 2.ntw.008`` | Device | hosting | .013 `__ | n-on-bare-m | | | | SR-IOV | | etal-infrat | | | | ac | | ructure>`__ | | | | celeration, | | | | | | a Device | | | | | | Plugin for | | | | | | SR-IOV must | | | | | | be used to | | | | | | configure | | | | | | the SR-IOV | | | | | | devices and | | | | | | advertise | | | | | | them to the | | | | | | ` | | | | | | `kubelet``. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Multiple | When a | `req.ge | `4 | | 2.ntw.009`` | connection | m | n.ost.01 `__ | installatio | | | | non-default | | n-on-bare-m | | | | connection | | etal-infrat | | | | points must | | ructure>`__ | | | | be managed | | | | | | by a | | | | | | CNI | | | | | | -conformant | | | | | | Network | | | | | | Plugin. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | User plane | When | `infra. | `4 | | 2.ntw.010`` | networking | hosting | net.acc.cfg | .3.1 `__ | etal-infrat | | | | that | | ructure>`__ | | | | support the | | | | | | use of | | | | | | DPDK, VPP, | | | | | | or SR-IOV | | | | | | must be | | | | | | deployed as | | | | | | part of the | | | | | | networking | | | | | | solution. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | NATless | When | `req.in | | | 2.ntw.011`` | c | hosting | f.ntw.14 `__ | | | | | IP | | | | | | addresses | | | | | | to be | | | | | | preserved | | | | | | in the | | | | | | traffic | | | | | | headers, a | | | | | | CNI plugin | | | | | | that | | | | | | exposes the | | | | | | pod IP | | | | | | directly to | | | | | | the | | | | | | external | | | | | | networks | | | | | | (e.g. | | | | | | Calico, | | | | | | MACVLAN or | | | | | | IPVLAN CNI | | | | | | plugins) is | | | | | | required. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Device | When | `e.cap. | `4 | | 2.ntw.012`` | Plugins | hosting | 016 `__, | n-on-bare-m | | | | require the | `e.cap | etal-infrat | | | | use of | .013 `__ | | | | FPGA, | er02.md#221 | | | | | SR-IOV or | -cloud-infr | | | | | other | astructure- | | | | | A | software-pr | | | | | cceleration | ofile-capab | | | | | Hardware, a | ilities>`__ | | | | | Device | | | | | | Plugin for | | | | | | that FPGA | | | | | | or | | | | | | A | | | | | | cceleration | | | | | | Hardware | | | | | | must be | | | | | | used. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Dual stack | The | `req.in | | | 2.ntw.013`` | CNI | networking | f.ntw.04 `__ | | | | | must use a | | | | | | CNI | | | | | | -conformant | | | | | | Network | | | | | | Plugin that | | | | | | is able to | | | | | | support | | | | | | dual-stack | | | | | | IPv4/IPv6 | | | | | | networking. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Security | The | `in | | | 2.ntw.014`` | Groups | networking | fra.net.cfg | | | | | solution | .004 `__ | | | | | network | | | | | | policies. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | IPAM plugin | When a | `req.in | | | 2.ntw.015`` | for | m | f.ntw.10 `__ | | | | | IPAM | | | | | | Network | | | | | | Plugin | | | | | | **must** be | | | | | | installed | | | | | | to allocate | | | | | | IP | | | | | | addresses | | | | | | for | | | | | | secondary | | | | | | network | | | | | | interfaces | | | | | | across all | | | | | | nodes of | | | | | | the | | | | | | cluster. | | | +-------------+-------------+-------------+-------------+-------------+ .. raw:: html

Table 4-4: Networking Solution Specifications

.. _46-storage-components: 4.6 Storage components ---------------------- In order for the storage solution(s) to be conformant with the Reference Architecture they must be implemented as per the following specifications: +-------------+-------------+-------------+-------------+-------------+ | Ref | Sp | Details | Requirement | Reference | | | ecification | | Trace | Imp | | | | | | lementation | | | | | | Trace | +=============+=============+=============+=============+=============+ | ``ra | Ephemeral | An | | | | 2.stg.001`` | Storage | imp | | | | | | lementation | | | | | | must | | | | | | support | | | | | | ephemeral | | | | | | storage, | | | | | | for the | | | | | | unpacked | | | | | | container | | | | | | images to | | | | | | be stored | | | | | | and | | | | | | executed | | | | | | from, as a | | | | | | directory | | | | | | in the | | | | | | filesystem | | | | | | on the | | | | | | worker node | | | | | | on which | | | | | | the | | | | | | container | | | | | | is running. | | | | | | See the | | | | | | `Container | | | | | | runtime | | | | | | s <#4.4>`__ | | | | | | section | | | | | | above for | | | | | | more | | | | | | information | | | | | | on how this | | | | | | meets the | | | | | | requirement | | | | | | for | | | | | | ephemeral | | | | | | storage for | | | | | | containers. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Kubernetes | An | | | | 2.stg.002`` | Volumes | imp | | | | | | lementation | | | | | | may attach | | | | | | additional | | | | | | storage to | | | | | | containers | | | | | | using | | | | | | Kubernetes | | | | | | Volumes. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Kubernetes | An | | | | 2.stg.003`` | Volumes | imp | | | | | | lementation | | | | | | may use | | | | | | Volume | | | | | | Plugins | | | | | | (see | | | | | | ``ra | | | | | | 2.stg.005`` | | | | | | below) to | | | | | | allow the | | | | | | use of a | | | | | | storage | | | | | | protocol | | | | | | (e.g. | | | | | | iSCSI, NFS) | | | | | | or | | | | | | management | | | | | | API (e.g. | | | | | | Cinder, | | | | | | EBS) for | | | | | | the | | | | | | attaching | | | | | | and | | | | | | mounting of | | | | | | storage | | | | | | into a Pod. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Persistent | An | `req.in | | | 2.stg.004`` | Volumes | imp | f.stg.01 `__ | | | | | (PV) to | | | | | | provide | | | | | | persistent | | | | | | storage for | | | | | | Pods | | | | | | .Persistent | | | | | | Volumes | | | | | | exist | | | | | | independent | | | | | | of the | | | | | | lifecycle | | | | | | of | | | | | | containers | | | | | | and/or | | | | | | pods. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Storage | Volume | | | | 2.stg.005`` | Extension | plugins | | | | | | must allow | | | | | | for the use | | | | | | of a range | | | | | | of backend | | | | | | storage | | | | | | systems. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Container | An | | | | 2.stg.006`` | Storage | imp | | | | | Interface | lementation | | | | | (CSI) | may support | | | | | | the | | | | | | Container | | | | | | Storage | | | | | | Interface | | | | | | (CSI), an | | | | | | Out-of-tree | | | | | | plugin.In | | | | | | order to | | | | | | support | | | | | | CSI, the | | | | | | feature | | | | | | gates | | | | | | ``CSIDrive | | | | | | rRegistry`` | | | | | | and | | | | | | ``CS | | | | | | INodeInfo`` | | | | | | must be | | | | | | enabled.The | | | | | | imp | | | | | | lementation | | | | | | must use a | | | | | | CSI driver | | | | | | (a full | | | | | | list of CSI | | | | | | drivers can | | | | | | be found | | | | | | `here `__). | | | | | | An | | | | | | imp | | | | | | lementation | | | | | | may support | | | | | | ephemeral | | | | | | storage | | | | | | through a | | | | | | CSI | | | | | | -compatible | | | | | | volume | | | | | | plugin in | | | | | | which case | | | | | | the | | | | | | ``CSIInl | | | | | | ineVolume`` | | | | | | feature | | | | | | gate must | | | | | | be | | | | | | enabled.An | | | | | | imp | | | | | | lementation | | | | | | may support | | | | | | Persistent | | | | | | Volumes | | | | | | through a | | | | | | CSI | | | | | | -compatible | | | | | | volume | | | | | | plugin in | | | | | | which case | | | | | | the | | | | | | ` | | | | | | `CSIPersist | | | | | | entVolume`` | | | | | | feature | | | | | | gate must | | | | | | be enabled. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | | An | | | | 2.stg.007`` | | imp | | | | | | lementation | | | | | | should use | | | | | | Kubernetes | | | | | | Storage | | | | | | Classes to | | | | | | support | | | | | | automation | | | | | | and the | | | | | | separation | | | | | | of concerns | | | | | | between | | | | | | providers | | | | | | of a | | | | | | service and | | | | | | consumers | | | | | | of the | | | | | | service. | | | +-------------+-------------+-------------+-------------+-------------+ .. raw:: html

Table 4-6: Storage Solution Specifications

A note on object storage: - This Reference Architecture does not include any specifications for object storage, as this is neither a native Kubernetes object, nor something that is required by CSI drivers. Object storage is an application-level requirement that would ordinarily be provided by a highly scalable service offering rather than being something an individual Kubernetes Cluster could offer. .. Todo: specifications/commentary to support req.inf.stg.04 (SDS) and req.inf.stg.05 (high performance and horizontally scalable storage). Also req.sec.gen.06 (storage resource isolation), req.sec.gen.10 (CIS - if applicable) and req.sec.zon.03 (data encryption at rest). .. _47-service-meshes: 4.7 Service meshes ------------------ Application service meshes are not in scope for the architecture. Network service mesh specifications are handled in section `4.5 Networking solutions <#45-networking-solutions>`__. .. _48-kubernetes-application-package-manager: 4.8 Kubernetes Application package manager ------------------------------------------ In order for the storage solution(s) to be conformant with the Reference Architecture they must be implemented as per the following specifications: +-------------+-------------+-------------+-------------+-------------+ | Ref | Sp | Details | Requirement | Reference | | | ecification | | Trace | Imp | | | | | | lementation | | | | | | Trace | +=============+=============+=============+=============+=============+ | ``ra | API-based | A package | `req.int. | | | 2.pkg.001`` | package | manager | api.02 <./c | | | | management | must use | hapter02.md | | | | | the | #23-kuberne | | | | | Kubernetes | tes-archite | | | | | APIs to | cture-requi | | | | | manage | rements>`__ | | | | | application | | | | | | artefacts. | | | | | | C | | | | | | luster-side | | | | | | components | | | | | | such as | | | | | | Tiller are | | | | | | not | | | | | | supported. | | | +-------------+-------------+-------------+-------------+-------------+ .. raw:: html

Table 4-7: Kubernetes Application Package Management Specifications

.. _49-kubernetes-workloads: 4.9 Kubernetes workloads ------------------------ In order for the Kubernetes workloads to be conformant with the Reference Architecture they must be implemented as per the following specifications: +-------------+-------------+-------------+-------------+-------------+ | Ref | Sp | Details | Requirement | Reference | | | ecification | | Trace | Imp | | | | | | lementation | | | | | | Trace | +=============+=============+=============+=============+=============+ | ``ra | `R | Specifies | TBD | N/A | | 2.app.001`` | oot `__ | | | | | | Parameter | | | | | | Group (OCI | | | | | | Spec) | | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | `Mounts `__ | | | | | | Parameter | | | | | | Group (OCI | | | | | | Spec) | | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | `P | Specifies | TBD | N/A | | 2.app.003`` | rocess `__ | | | | | | Parameter | | | | | | Group (OCI | | | | | | Spec) | | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | `Hos | Specifies | TBD | N/A | | 2.app.004`` | tname `__ | container | | | | | Parameter | | | | | | Group (OCI | | | | | | Spec) | | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | `User < | User for | TBD | N/A | | 2.app.005`` | https://git | the process | | | | | hub.com/ope | is a | | | | | ncontainers | platfo | | | | | /runtime-sp | rm-specific | | | | | ec/blob/mas | structure | | | | | ter/config. | that allows | | | | | md#user>`__ | specific | | | | | Parameter | control | | | | | Group (OCI | over which | | | | | Spec) | user the | | | | | | process | | | | | | runs as | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Consumption | The | `req.in | N/A | | 2.app.006`` | of | workload | t.api.01 `__ | | | | | points | | | | | | through the | | | | | | use of | | | | | | workload | | | | | | annotations | | | | | | or resource | | | | | | requests | | | | | | and limits | | | | | | within the | | | | | | container | | | | | | spec passed | | | | | | to the | | | | | | Kubernetes | | | | | | API Server. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Host | Workloads | `req.kc | N/A | | 2.app.007`` | Volumes | should not | m.gen.02 `__ | | | | | identical | | | | | | co | | | | | | nfiguration | | | | | | (such as | | | | | | created | | | | | | from a | | | | | | P | | | | | | odTemplate) | | | | | | may behave | | | | | | differently | | | | | | on | | | | | | different | | | | | | nodes due | | | | | | to | | | | | | different | | | | | | files on | | | | | | the | | | | | | nodes. | | | | | | `__ | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Inf | Workloads | TBD | N/A | | 2.app.008`` | rastructure | must not | | | | | dependency | rely on the | | | | | | a | | | | | | vailability | | | | | | of the | | | | | | master | | | | | | nodes for | | | | | | the | | | | | | successful | | | | | | execution | | | | | | of their | | | | | | fu | | | | | | nctionality | | | | | | (i.e. loss | | | | | | of the | | | | | | master | | | | | | nodes may | | | | | | affect | | | | | | non | | | | | | -functional | | | | | | behaviours | | | | | | such as | | | | | | healing and | | | | | | scaling, | | | | | | but | | | | | | components | | | | | | that are | | | | | | already | | | | | | running | | | | | | will | | | | | | continue to | | | | | | do so | | | | | | without | | | | | | issue). | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Device | Workload | TBD | N/A | | 2.app.009`` | plugins | descriptors | | | | | | must use | | | | | | the | | | | | | resources | | | | | | advertised | | | | | | by the | | | | | | device | | | | | | plugins to | | | | | | indicate | | | | | | their need | | | | | | for an | | | | | | FPGA, | | | | | | SR-IOV or | | | | | | other | | | | | | a | | | | | | cceleration | | | | | | device. | | | +-------------+-------------+-------------+-------------+-------------+ | ``ra | Node | Workload | TBD | N/A | | 2.app.010`` | Feature | descriptors | | | | | Discovery | must use | | | | | (NFD) | the labels | | | | | | advertised | | | | | | by `Node | | | | | | Feature | | | | | | Di | | | | | | scovery `__ | | | | | | to indicate | | | | | | which node | | | | | | software of | | | | | | hardware | | | | | | features | | | | | | they need. | | | +-------------+-------------+-------------+-------------+-------------+ .. raw:: html

Table 4-8: Kubernetes Workload Specifications

.. _410-additional-required-components: 4.10 Additional required components ----------------------------------- This chapter should list any additional components needed to provide the services defined in Chapter 3.2 (e.g: Prometheus)