doc-exports/docs/cce/umn/cce_bulletin_0059.html

<a name="cce_bulletin_0059"></a><a name="cce_bulletin_0059"></a>

<h1 class="topictitle1">Kubernetes 1.27 Release Notes</h1>
<div id="body8662426"><p id="cce_bulletin_0059__en-us_topic_0000001649273541_en-us_topic_0000001072975092_en-us_topic_0261805759_en-us_topic_0261793154_p8060118">CCE now supports Kubernetes 1.27 cluster features. This section describes the changes made in Kubernetes 1.27 compared to Kubernetes 1.25.</p>
<div class="section" id="cce_bulletin_0059__en-us_topic_0000001649273541_section1391012495239"><h4 class="sectiontitle">Indexes</h4><ul id="cce_bulletin_0059__en-us_topic_0000001649273541_ul8733155692314"><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li17733195672318"><a href="#cce_bulletin_0059__en-us_topic_0000001649273541_section14972102702312">New Features</a></li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li2606144132418"><a href="#cce_bulletin_0059__en-us_topic_0000001649273541_section1898982110241">Deprecations and Removals</a></li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li1036516205388"><a href="#cce_bulletin_0059__en-us_topic_0000001649273541_section115291322132513">Enhanced Kubernetes 1.27 on CCE</a></li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li1076716133914"><a href="#cce_bulletin_0059__en-us_topic_0000001649273541_en-us_topic_0000001072975092_en-us_topic_0261805759_en-us_topic_0261793154_section1272182810583">References</a></li></ul>
</div>
<div class="section" id="cce_bulletin_0059__en-us_topic_0000001649273541_section14972102702312"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_section14972102702312"></a><a name="en-us_topic_0000001649273541_section14972102702312"></a><h4 class="sectiontitle">New Features</h4><div class="p" id="cce_bulletin_0059__en-us_topic_0000001649273541_p1404335230"><strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b16877105193917">Kubernetes 1.27</strong><ul id="cce_bulletin_0059__en-us_topic_0000001649273541_ul15117105663918"><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li693144814507">SeccompDefault is stable.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p877365712152"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li693144814507"></a><a name="en-us_topic_0000001649273541_li693144814507"></a>To use SeccompDefault, add the <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b396810398919">--seccomp-default</strong> <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/" target="_blank" rel="noopener noreferrer">command line flag</a> using kubelet on each node. If this feature is enabled, the <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b67611713481">RuntimeDefault</strong> profile will be used for all workloads by default, instead of the <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b137781440114314">Unconfined</strong> (seccomp disabled) profile.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li1534122119571">Jobs' scheduling directives are configurable.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p57089355217"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li1534122119571"></a><a name="en-us_topic_0000001649273541_li1534122119571"></a>This feature was introduced in Kubernetes 1.22 and is stable in Kubernetes 1.27. In most cases, you use a job to influence where the pods will run, like all in the same AZ. This feature allows modifying scheduling directives before a job starts. You can suspend a job using the <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b468301412148">suspend</strong> field. In the suspension phase, the scheduling directives (such as the node selector, node affinity, anti-affinity, and tolerations) in the job's pod template can be modified. For details, see <a href="https://kubernetes.io/docs/concepts/workloads/controllers/job/#mutable-scheduling-directives" target="_blank" rel="noopener noreferrer">Mutable Scheduling Directives</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li146636172135">Downward API hugepages are stable.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p19723131157"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li146636172135"></a><a name="en-us_topic_0000001649273541_li146636172135"></a>In Kubernetes 1.20, <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b85261900469">requests.hugepages-</strong><em id="cce_bulletin_0059__en-us_topic_0000001649273541_i7642311469">&lt;pagesize&gt;</em> and <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b1569511412462">limits.hugepages-</strong><em id="cce_bulletin_0059__en-us_topic_0000001649273541_i1364881734619">&lt;pagesize&gt;</em> were introduced to the <a href="https://kubernetes.io/docs/concepts/workloads/pods/downward-api/" target="_blank" rel="noopener noreferrer">downward API</a>. Requests and limits can be configured for hugepages like other resources.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li119828274139">Pod scheduling readiness moves to beta.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p13462545189"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li119828274139"></a><a name="en-us_topic_0000001649273541_li119828274139"></a>After a pod is created, the Kubernetes scheduler selects an appropriate node to run the pod in the pending state. In practice, some pods may stay in the pending state for a long period due to insufficient resources. These pods may affect the running of other components like Cluster Autoscaler in the cluster. By specifying or deleting <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b923261216383">.spec. schedulingGates</strong> for a pod, you can control when the pod is ready for scheduling. For details, see <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/pod-scheduling-readiness/" target="_blank" rel="noopener noreferrer">Pod Scheduling Readiness</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li160094718131">Accessing node logs using Kubernetes APIs is supported.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p26069499145"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li160094718131"></a><a name="en-us_topic_0000001649273541_li160094718131"></a>This function is in the alpha phase. The cluster administrator can directly query node logs to help debug malfunctioning services running on the node. To use this function, ensure that the NodeLogQuery <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/" target="_blank" rel="noopener noreferrer">feature gate</a> is enabled for that node and the kubelet configuration options <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b1626914357429">enableSystemLogHandler</strong> and <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b13273123617421">enableSystemLogQuery</strong> are set to <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b1786012447425">true</strong>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li46271858201310">ReadWriteOncePod access mode moves to beta.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p6361720101814"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li46271858201310"></a><a name="en-us_topic_0000001649273541_li46271858201310"></a>Kubernetes 1.22 introduced a ReadWriteOncePod access mode for PVs and PVCs. This feature has evolved into the beta phase. A volume can be mounted to a single pod in read/write mode. Use this access mode if you want to ensure that only one pod in the cluster can read that PVC or write to it. For details, see <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes" target="_blank" rel="noopener noreferrer">Access Modes</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li782110101146">The <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b318040152115">matchLabelKeys</strong> field in the pod topology spread constraint moves to beta.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p1267014962219"><strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b8396141313230">matchLabelKeys</strong> is a list of pod label keys. It is used to select a group of pods over which spreading will be calculated. With <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b9992641142513">matchLabelKeys</strong>, you do not need to update <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b20602135212250">pod.spec</strong> between different revisions. The controller or operator just needs to set different values to the same label key for different revisions. The scheduler will automatically determine the values based on <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b15321819132718">matchLabelKeys</strong>. For details, see <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#topologyspreadconstraints-field" target="_blank" rel="noopener noreferrer">Pod Topology Distribution Constraints</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li168762391144">The function of efficiently labeling SELinux volumes moves to beta.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p6951934162818"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li168762391144"></a><a name="en-us_topic_0000001649273541_li168762391144"></a>By default, the container runtime recursively assigns the SELinux label to all files in all pod volumes. To speed up this process, Kubernetes uses the mount option <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b1634505114498">-o context=</strong><em id="cce_bulletin_0059__en-us_topic_0000001649273541_i5490953184920">&lt;label&gt;</em> to immediately set the SELinux label of the volume. For details, see <a href="https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#efficient-selinux-volume-relabeling" target="_blank" rel="noopener noreferrer">Efficient SELinux volume relabeling</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li637192816398">VolumeManager reconstruction goes to beta.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p1769141115329"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li637192816398"></a><a name="en-us_topic_0000001649273541_li637192816398"></a>After the VolumeManager is reconstructed, if the NewVolumeManagerReconstruction <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/" target="_blank" rel="noopener noreferrer">feature gate</a> is enabled, mounted volumes will be obtained more effectively during kubelet startup.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li1323343518566">Server side field validation and OpenAPI V3 are stable.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p192898273210"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li1323343518566"></a><a name="en-us_topic_0000001649273541_li1323343518566"></a>OpenAPI V3 was added to Kubernetes 1.23. In Kubernetes 1.24, it moved to beta. In Kubernetes 1.27, it is stable.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li9978119205711">StatefulSet start ordinals move to beta.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p112716216572"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li9978119205711"></a><a name="en-us_topic_0000001649273541_li9978119205711"></a>Kubernetes 1.26 introduced a new, alpha-level feature for StatefulSets to control the ordinal numbering of pod replicas. Since Kubernetes 1.27, this feature moves to beta. The ordinals can start from arbitrary non-negative numbers. For details, see <a href="https://kubernetes.io/blog/2023/04/28/statefulset-start-ordinal/" target="_blank" rel="noopener noreferrer">Kubernetes 1.27: StatefulSet Start Ordinal Simplifies Migration</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li54582411183"><strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b192671952171114">ContainerResource</strong> metric in HorizontalPodAutoscaler moves to beta.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p29298831312">Kubernetes 1.20 introduced <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b20817440171117"><a href="https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/#container-resource-metrics" target="_blank" rel="noopener noreferrer">container resource metrics</a></strong> in Horizontal Pod Autoscalers (HPAs). In Kubernetes 1.27, this feature moves to beta, and the HPAContainerMetrics feature gate is enabled by default.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li3732125107">StatefulSet PVC auto deletion moves to beta.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p1433533781415"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li3732125107"></a><a name="en-us_topic_0000001649273541_li3732125107"></a>Kubernetes 1.27 provides a new policy to control the lifecycle of PVCs of StatefulSets. This policy allows users to specify whether the PVCs generated from the StatefulSet spec template should be automatically deleted or retained when the StatefulSet is deleted or replicas in the StatefulSet are scaled down. For details, see <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#persistentvolumeclaim-retention" target="_blank" rel="noopener noreferrer">PersistentVolumeClaim retention</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li16252171511106">Volume group snapshots are introduced.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p1454172701711"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li16252171511106"></a><a name="en-us_topic_0000001649273541_li16252171511106"></a>Volume group snapshots are introduced as an alpha feature in Kubernetes 1.27. This feature allows users to create snapshots for multiple volumes to ensure data consistency when a fault occurs. It uses a label selector to group multiple PVCs for snapshots. This feature only supports CSI volume drivers. For details, see <a href="https://kubernetes.io/blog/2023/05/08/kubernetes-1-27-volume-group-snapshot-alpha/" target="_blank" rel="noopener noreferrer">Kubernetes 1.27: Introducing an API for Volume Group Snapshots</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li55071727181017"><strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b29751727122110">kubectl apply</strong> pruning is more secure and efficient.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p674119222293">In Kubernetes 1.5, the <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b868385892111">--prune</strong> flag was introduced in <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b14303177172212">kubectl apply</strong> to delete resources that are no longer needed. This allowed <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b560643917229">kubectl apply</strong> to automatically clear resources removed from the current configuration. However, the existing implementation of <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b11942155011221">--prune</strong> has design defects that degrade its performance and lead to unexpected behaviors. In Kubernetes 1.27, <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b193253215255">kubectl apply</strong> provides ApplySet-based pruning, which is in the alpha phase. For details, see <a href="https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/#alternative-kubectl-apply-f-directory-prune" target="_blank" rel="noopener noreferrer">Declarative Management of Kubernetes Objects Using Configuration Files</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li396944019103">Conflicts during port allocation to NodePort Service can be avoided.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p5723175914239"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li396944019103"></a><a name="en-us_topic_0000001649273541_li396944019103"></a>In Kubernetes 1.27, you can enable a new <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/" target="_blank" rel="noopener noreferrer">feature gate </a>ServiceNodePortStaticSubrange to use different port allocation policies for NodePort Services. This mitigates the risk of port conflicts. This feature is in the alpha phase.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li103971949201013">Resizing resources assigned to pods without restarting the containers is supported.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p113301182416"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li103971949201013"></a><a name="en-us_topic_0000001649273541_li103971949201013"></a>Kubernetes 1.27 allows users to resize CPU and memory resources assigned to pods without restarting the container. This feature is in the alpha phase. For details, see <a href="https://kubernetes.io/blog/2023/05/12/in-place-pod-resize-alpha/" target="_blank" rel="noopener noreferrer">Kubernetes 1.27: In-place Resource Resize for Kubernetes Pods (alpha)</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li34551755111012">Pod startup is accelerated.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p162748262414"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li34551755111012"></a><a name="en-us_topic_0000001649273541_li34551755111012"></a>A series of parameter adjustments like parallel image pulls and increased default API query limit for kubelet per second are made in Kubernetes 1.27 to accelerate pod startup. For details, see <a href="https://kubernetes.io/blog/2023/05/15/speed-up-pod-startup/" target="_blank" rel="noopener noreferrer">Kubernetes 1.27: updates on speeding up Pod startup</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li2517105111118">KMS V2 moves to beta.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p1111114818231"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li2517105111118"></a><a name="en-us_topic_0000001649273541_li2517105111118"></a>The key management KMS V2 API goes to beta. This has greatly improved the performance of the KMS encryption provider. For details, see <a href="https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/" target="_blank" rel="noopener noreferrer">Using a KMS provider for data encryption</a>.</p>
</li></ul>
</div>
<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p1553515119414"><strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b14535811174115">Kubernetes 1.26</strong></p>
<ul id="cce_bulletin_0059__en-us_topic_0000001649273541_ul18736151984212"><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li105611555154914">CRI v1alpha2 is removed.<div class="p" id="cce_bulletin_0059__en-us_topic_0000001649273541_p1853165674913"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li105611555154914"></a><a name="en-us_topic_0000001649273541_li105611555154914"></a>Kubernetes 1.26 does not support CRI v1alpha2. Use CRI v1 (containerd version must be later than or equal to 1.5.0). containerd 1.5.x or earlier is not supported by Kubernetes 1.26. Update the containerd version to 1.6.x or later before upgrading kubelet to 1.26.<div class="note" id="cce_bulletin_0059__en-us_topic_0000001649273541_note18338124411"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="cce_bulletin_0059__en-us_topic_0000001649273541_p14318816449">The containerd version used by CCE is 1.6.14, which meets the requirements. If the existing nodes do not meet the containerd version requirements, reset them to the latest version.</p>
</div></div>
</div>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li17737161912424">Alpha API for dynamic resource allocation is added.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p1235143312517"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li17737161912424"></a><a name="en-us_topic_0000001649273541_li17737161912424"></a>In Kubernetes 1.26, <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/" target="_blank" rel="noopener noreferrer">Dynamic Resource Allocation</a> is added to request and share resources between pods and between containers in a pod. Resources are initialized based on parameters provided by the user. This function is still in the alpha phase. You need to enable the DynamicResourceAllocation feature gate and the <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b19871149195715">resource.k8s.io/v1alpha1</strong> API group. You need to install drivers for specific resources to be managed. For details, see <a href="https://kubernetes.io/blog/2022/12/15/dynamic-resource-allocation/" target="_blank" rel="noopener noreferrer">Kubernetes 1.26: Alpha API for Dynamic Resource Allocation</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li1792818161614">The non-graceful node shutdown feature goes to beta.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p139501942152316"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li1792818161614"></a><a name="en-us_topic_0000001649273541_li1792818161614"></a>In Kubernetes 1.26, the non-graceful node shutdown feature goes to beta and is enabled by default. A node shutdown can be graceful only if the kubelet's node shutdown manager can detect the upcoming node shutdown action. For details, see <a href="https://kubernetes.io/docs/concepts/architecture/nodes/#non-graceful-node-shutdown" target="_blank" rel="noopener noreferrer">Non-graceful node shutdown handling</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li13687171529">Passing pod <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b898218537416">fsGroup</strong> to CSI drivers during mounting is supported.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p183711740102112">In Kubernetes 1.22, delegation of <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b950951818618">fsGroup</strong> to CSI drivers was first introduced as an alpha feature. In Kubernetes 1.25, it moved to beta. In Kubernetes 1.26, this feature enters the official release phase. For details, see <a href="https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#delegating-volume-permission-and-ownership-change-to-csi-driver" target="_blank" rel="noopener noreferrer">Delegating volume permission and ownership change to CSI driver</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li19898131811215">Pod scheduling readiness is introduced.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p19967118152116"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li19898131811215"></a><a name="en-us_topic_0000001649273541_li19898131811215"></a>Kubernetes 1.26 introduces a new feature schedulingGates, which enables the scheduler to be aware of when pod scheduling can be performed. For details, see <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/pod-scheduling-readiness/" target="_blank" rel="noopener noreferrer">Pod Scheduling Readiness</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li25118411821">CPU manager is officially released.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p648191514166"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li25118411821"></a><a name="en-us_topic_0000001649273541_li25118411821"></a>The CPU manager is a part of kubelet. Since Kubernetes 1.10, it has moved to <a href="https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-manager/" target="_blank" rel="noopener noreferrer">beta</a>. The CPU manager can allocate exclusive CPUs to containers. This feature is stable in Kubernetes 1.26. For details, see <a href="https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/" target="_blank" rel="noopener noreferrer">Control CPU Management Policies on the Node</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li01268535217">Kubernetes traffic engineering is advanced.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p66561142114"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li01268535217"></a><a name="en-us_topic_0000001649273541_li01268535217"></a><a href="https://kubernetes.io/blog/2022/12/30/advancements-in-kubernetes-traffic-engineering/#optimizing-internal-node-local-traffic" target="_blank" rel="noopener noreferrer">Internal node-local traffic optimization</a> and <a href="https://kubernetes.io/blog/2022/12/30/advancements-in-kubernetes-traffic-engineering/#endpointslice-conditions" target="_blank" rel="noopener noreferrer">EndpointSlice conditions</a> are upgraded to the official release version. <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b2103614247"><a href="https://kubernetes.io/blog/2022/12/30/advancements-in-kubernetes-traffic-engineering/#traffic-loss-from-load-balancers-during-rolling-updates" target="_blank" rel="noopener noreferrer">ProxyTerminatingEndpoints</a></strong> moves to beta.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li97756318310">Cross-namespace volume data sources are supported.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p5269521594"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li97756318310"></a><a name="en-us_topic_0000001649273541_li97756318310"></a>This feature allows you to specify a data source that belongs to different namespaces for a PVC. This feature is in the alpha phase. For details, see <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#cross-namespace-data-sources" target="_blank" rel="noopener noreferrer">Cross namespace data sources</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li148028171133">Retroactive default StorageClass assignment moves to beta.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p16481537166"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li148028171133"></a><a name="en-us_topic_0000001649273541_li148028171133"></a>In Kubernetes 1.25, an alpha feature was introduced to change the way a default StorageClass is allocated to a PVC. After this feature is enabled, you no longer need to create a default StorageClass and then create a PVC to assign the class. Additionally, any PVCs without a StorageClass assigned can be updated later. This feature moves to beta in Kubernetes 1.26. For details, see <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#retroactive-default-storageclass-assignment" target="_blank" rel="noopener noreferrer">Retroactive default StorageClass assignment</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li153051735235">PodDisruptionBudget allows users to specify the eviction policies for unhealthy pods.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p375262515417"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li153051735235"></a><a name="en-us_topic_0000001649273541_li153051735235"></a>You are allowed to specify unhealthy pod eviction policies for <a href="https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets" target="_blank" rel="noopener noreferrer">PodDisruptionBudget</a> (PDB). This feature helps ensure node availability during node management. This feature is in the beta phase. For details, see <a href="https://kubernetes.io/docs/tasks/run-application/configure-pdb/#unhealthy-pod-eviction-policy" target="_blank" rel="noopener noreferrer">Unhealthy Pod Eviction Policy</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li19697122414273">The number of Horizontal Pod Autoscalers (HPAs) can be configured.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p12541162642712"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li19697122414273"></a><a name="en-us_topic_0000001649273541_li19697122414273"></a><strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b512419174111">kube-controller-manager</strong> allows <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b549453717120">--concurrent-horizontal-pod-autoscaler-syncs</strong> to configure the number of worker nodes of the pod autoscaler for horizontal scaling. </p>
</li></ul>
</div>
<div class="section" id="cce_bulletin_0059__en-us_topic_0000001649273541_section1898982110241"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_section1898982110241"></a><a name="en-us_topic_0000001649273541_section1898982110241"></a><h4 class="sectiontitle">Deprecations and Removals</h4><p id="cce_bulletin_0059__en-us_topic_0000001649273541_p1624951055111"><strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b1653632255111">Kubernetes 1.27</strong></p>
<ul id="cce_bulletin_0059__en-us_topic_0000001649273541_ul8728123509"><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li81879568511">In Kubernetes 1.27, the feature gates that are used for volume extension and in the GA status, including ExpandCSIVolumes, ExpandInUsePersistentVolumes, and ExpandPersistentVolumes, are removed and can no longer be referenced in the <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b08390125913">--feature-gates</strong> flag.</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li4187175611518">The <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b13186121111317">--master-service-namespace</strong> parameter is removed. This parameter specifies where to create a Service named <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b962253601314">kubernetes</strong> to represent the API server. This parameter was deprecated in Kubernetes 1.26 and is removed from Kubernetes 1.27.</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li101871956165112">The ControllerManagerLeaderMigration feature gate is removed. <a href="https://github.com/kubernetes/enhancements/issues/2436" target="_blank" rel="noopener noreferrer">Leader Migration</a> provides a mechanism for HA clusters to safely migrate "cloud specific" controllers using a resource lock shared between kube-controller-manager and cloud-controller-manager when upgrading the replicated control plane. This feature has been enabled unconditionally since its release in Kubernetes 1.24. In Kubernetes 1.27, this feature is removed.</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li191871156195118">The <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b156028250592">--enable-taint-manager</strong> parameter is removed. The feature that it supports, taint-based eviction, is enabled by default. It will continue to be implicitly enabled when the flag is removed.</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li18187165655119">The <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b1840552715533">--pod-eviction-timeout</strong> parameter is removed from kube-controller-manager.</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li518715605116">The CSIMigration feature gate is removed. The <a href="https://github.com/kubernetes/enhancements/issues/625" target="_blank" rel="noopener noreferrer">CSI migration</a> program allows smooth migration from the in-tree volume plug-ins to the out-of-tree CSI drivers. This feature was officially released in Kubernetes 1.16.</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li17187195665118">The CSIInlineVolume feature gate is removed. The feature (<a href="https://github.com/kubernetes/kubernetes/pull/111258" target="_blank" rel="noopener noreferrer">CSI Ephemeral Volume</a>) allows CSI volumes to be specified directly in the pod specification for ephemeral use cases. They can be used to inject arbitrary states, such as configuration, secrets, identity, variables, or similar information, directly inside the pod using a mounted volume. This feature graduated to GA in Kubernetes 1.25 and is removed in Kubernetes 1.27.</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li1018716562516">The EphemeralContainers feature gate is removed. For Kubernetes 1.27, API support for ephemeral containers is unconditionally enabled.</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li12187105611514">The LocalStorageCapacityIsolation feature gate is removed. This feature gate (<a href="https://github.com/kubernetes/kubernetes/pull/111513" target="_blank" rel="noopener noreferrer">Local Ephemeral Storage Capacity Isolation</a>) moved to GA in Kubernetes 1.25. The feature provides support for capacity isolation of local ephemeral storage between pods, such as emptyDir volumes, so that a pod can be limited in its consumption of shared resources. kubelet will evict a pod if its consumption of local ephemeral storage exceeds the configured limit.</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li11187115614519">The NetworkPolicyEndPort feature gate is removed. In Kubernetes 1.25, <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b144710544010">endPort</strong> in NetworkPolicy moved to GA. NetworkPolicy providers that support the <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b15486566017">endPort</strong> field can be used to specify a range of ports to apply NetworkPolicy.</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li1918712560510">The StatefulSetMinReadySeconds feature gate is removed. For a pod that is part of a StatefulSet, Kubernetes marks the pod as read-only when the pod is available (and passes the check) for at least within the period specified in <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b196156413"><a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#minimum-ready-seconds" target="_blank" rel="noopener noreferrer">minReadySeconds</a></strong>. This feature was officially released in Kubernetes 1.25. It is locked to <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b332717591717">true</strong> and removed from Kubernetes 1.27.</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li2187155675118">The IdentifyPodOS feature gate is removed. If this feature is enabled, you can specify an OS for a pod. It has been stable since Kubernetes 1.25. This feature is removed from Kubernetes 1.27.</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li2018745619517">The DaemonSetUpdateSurge feature gate is removed. In Kubernetes 1.25, this feature was stable. It was implemented to minimize DaemonSet downtime during deployment, but it is removed from Kubernetes 1.27.</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li11871656175115">The <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b16937518520">--container-runtime</strong> parameter is removed. kubelet accepts a deprecated parameter <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b122212214">--container-runtime</strong>. After the dockershim code is removed, the only valid value for this parameter will be <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b1324211321">remote</strong>. This parameter was deprecated in 1.24 and later versions and is removed from Kubernetes 1.27.</li></ul>
<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p1938218322522"><strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b386654685217">Kubernetes 1.26</strong></p>
<ul id="cce_bulletin_0059__en-us_topic_0000001649273541_ul12827723111"><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li9792154813612">HorizontalPodAutoscaler API for v2beta2 is removed.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p42481850113614"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li9792154813612"></a><a name="en-us_topic_0000001649273541_li9792154813612"></a>The autoscaling/v2beta2 API of HorizontalPodAutoscaler is no longer available in Kubernetes 1.26. For details, see <a href="https://kubernetes.io/docs/reference/using-api/deprecation-guide/#horizontalpodautoscaler-v126" target="_blank" rel="noopener noreferrer">Removed APIs by release</a>. Use autoscaling/v2 API instead.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li1042182131110">The <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b35631718132617">flowcontrol.apiserver.k8s.io/v1beta1</strong> API is removed.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p149455154117">In Kubernetes 1.26 and later versions, the API of the <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b18692172143015">flowcontrol.apiserver.k8s.io/v1beta1</strong> version for FlowSchema and PriorityLevelConfiguration is no longer served. For details, see <a href="https://kubernetes.io/docs/reference/using-api/deprecation-guide/#horizontalpodautoscaler-v126" target="_blank" rel="noopener noreferrer">Removed APIs by release</a>. The <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b193421519183017">flowcontrol.apiserver.k8s.io/v1beta2</strong> version is available in Kubernetes 1.23 and later versions, and the <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b13570103215306">flowcontrol.apiserver.k8s.io/v1beta3</strong> version is available in Kubernetes 1.26 and later versions.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li4283875311">The cloud service vendors' in-tree storage drivers are removed.</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li728315719311">The kube-proxy userspace mode is removed.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p16394142214395"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li728315719311"></a><a name="en-us_topic_0000001649273541_li728315719311"></a>The deprecated userspace mode is no longer supported by Linux or Windows. Linux users can use Iptables or IPVS, and Windows users can use the Kernelspace mode. Errors are returned if you use <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b16591357361">--mode userspace</strong>.</p>
<ul id="cce_bulletin_0059__en-us_topic_0000001649273541_ul10283107193115"><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li1428397153117">Windows winkernel kube-proxy no longer supports Windows HNS v1 APIs.</li></ul>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li728313714319"><strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b17129155353611">--prune-whitelist</strong> flag is deprecated.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p028317783113">The <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b8550162163711">--prune-whitelist</strong> flag is <a href="https://github.com/kubernetes/kubernetes/pull/113116" target="_blank" rel="noopener noreferrer">deprecated</a> and replaced by <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b66811717113716">--prune-allowlist</strong> to support <a href="https://www.cncf.io/announcements/2021/10/13/inclusive-naming-initiative-announces-new-community-resources-for-a-more-inclusive-future/" target="_blank" rel="noopener noreferrer">Inclusive Naming Initiative</a>. This deprecated flag will be completely removed in later versions.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li9283127133112">The DynamicKubeletConfig feature gate is removed.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p102831719315"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li9283127133112"></a><a name="en-us_topic_0000001649273541_li9283127133112"></a>The kubelet configuration of nodes can be dynamically updated through the API. The feature gate is removed from the kubelet in Kubernetes 1.24 and removed from the API server in Kubernetes 1.26. This simplifies the code and improves stability. It is recommended that you modify the kubelet configuration file instead and then restart the kubelet. For details, see <a href="https://github.com/kubernetes/kubernetes/pull/112643" target="_blank" rel="noopener noreferrer">Remove DynamicKubeletConfig feature gate from the code</a>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li1841893084014">A kube-apiserver command line parameter is removed.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p1185644214018"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li1841893084014"></a><a name="en-us_topic_0000001649273541_li1841893084014"></a>The <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b13596112014418"><a href="https://github.com/kubernetes/kubernetes/pull/38186" target="_blank" rel="noopener noreferrer">--master-service-namespace</a></strong> parameter is deprecated. It is unused in the API Server.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li657118463401">Several <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b37239148128">kubectl run</strong> parameters are deprecated.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p7746547134014">Several unused kubectl subcommands are marked as <a href="https://github.com/kubernetes/kubernetes/pull/112261" target="_blank" rel="noopener noreferrer">deprecated</a> and will be removed in later versions. These subcommands include <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b145471605456">--cascade</strong>, <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b654612519457">--filename</strong>, <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b202331111104519">--force</strong>, <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b211551684515">--grace-period</strong>, <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b59242262451">--kustomize</strong>, <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b16684173218459">--recursive</strong>, <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b16124203704516">--timeout</strong>, and <strong id="cce_bulletin_0059__en-us_topic_0000001649273541_b7437114784515">--wait</strong>.</p>
</li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_li21829811421">Some command line parameters related to logging are removed.<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p1416611984218"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_li21829811421"></a><a name="en-us_topic_0000001649273541_li21829811421"></a>Some logging-related command line parameters are <a href="https://github.com/kubernetes/kubernetes/pull/112120" target="_blank" rel="noopener noreferrer">removed</a>. These parameters were <a href="https://github.com/kubernetes/enhancements/tree/3cb66bd0a1ef973ebcc974f935f0ac5cba9db4b2/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components#removed-klog-flags" target="_blank" rel="noopener noreferrer">deprecated</a> in earlier versions.</p>
</li></ul>
</div>
<div class="section" id="cce_bulletin_0059__en-us_topic_0000001649273541_section115291322132513"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_section115291322132513"></a><a name="en-us_topic_0000001649273541_section115291322132513"></a><h4 class="sectiontitle">Enhanced Kubernetes 1.27 on CCE</h4><p id="cce_bulletin_0059__en-us_topic_0000001649273541_p10835173722518">During a version maintenance period, CCE periodically updates Kubernetes 1.27 and provides enhanced functions.</p>
<p id="cce_bulletin_0059__en-us_topic_0000001649273541_p1752116404265">For details about cluster version updates, see <a href="cce_10_0405.html">Release Notes for CCE Cluster Versions</a>.</p>
</div>
<div class="section" id="cce_bulletin_0059__en-us_topic_0000001649273541_en-us_topic_0000001072975092_en-us_topic_0261805759_en-us_topic_0261793154_section1272182810583"><a name="cce_bulletin_0059__en-us_topic_0000001649273541_en-us_topic_0000001072975092_en-us_topic_0261805759_en-us_topic_0261793154_section1272182810583"></a><a name="en-us_topic_0000001649273541_en-us_topic_0000001072975092_en-us_topic_0261805759_en-us_topic_0261793154_section1272182810583"></a><h4 class="sectiontitle">References</h4><p id="cce_bulletin_0059__en-us_topic_0000001649273541_en-us_topic_0000001072975092_p15127239183119">For more details about the performance comparison and functional evolution between Kubernetes 1.27 and other versions, see the following documents:</p>
<ul id="cce_bulletin_0059__en-us_topic_0000001649273541_en-us_topic_0000001072975092_ul10912174853114"><li id="cce_bulletin_0059__en-us_topic_0000001649273541_en-us_topic_0000001072975092_li17912184843113"><a href="https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.27.md" target="_blank" rel="noopener noreferrer">Kubernetes v1.27 Release Notes</a></li><li id="cce_bulletin_0059__en-us_topic_0000001649273541_en-us_topic_0263124530_li6912194811317"><a href="https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.26.md" target="_blank" rel="noopener noreferrer">Kubernetes v1.26 Release Notes</a></li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="cce_10_0068.html">Kubernetes Version Release Notes</a></div>
</div>
</div>