forked from docs/doc-exports
qiujiandong1
6d480dcc20
Reviewed-by: Eotvos, Oliver <oliver.eotvos@t-systems.com> Co-authored-by: qiujiandong1 <qiujiandong1@huawei.com> Co-committed-by: qiujiandong1 <qiujiandong1@huawei.com>
124 lines
16 KiB
HTML
124 lines
16 KiB
HTML
<a name="cce_10_0848"></a><a name="cce_10_0848"></a>
|
|
|
|
<h1 class="topictitle1">Manually Upgrading the Driver Version of a GPU Node</h1>
|
|
<div id="body0000001827897644"><p id="cce_10_0848__p1850810436493">You can use the CCE AI Suite (NVIDIA GPU) add-on to configure the driver file path for a node. After the node is restarted, the driver will be installed automatically. Alternatively, manually upgrade the driver version.</p>
|
|
<div class="note" id="cce_10_0848__note6127173418403"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="cce_10_0848__p128878190474">Manually upgrading the driver version of a GPU node is a short-term fix for customizing node configurations. However, once the node is restarted, the driver version will revert to the version specified in the CCE AI Suite (NVIDIA GPU) settings.</p>
|
|
<p id="cce_10_0848__p789919514011">For a stable, long-term upgrade of the driver version of a GPU node, see <a href="cce_10_0849.html">Upgrading the Driver Version of a GPU Node Using a Node Pool</a>.</p>
|
|
</div></div>
|
|
<div class="section" id="cce_10_0848__section1038618563212"><h4 class="sectiontitle">Prerequisites</h4><p id="cce_10_0848__p13196104228">The cluster can be accessed using kubectl. For details, see <a href="cce_10_0107.html">Accessing a Cluster Using kubectl</a>.</p>
|
|
</div>
|
|
<div class="section" id="cce_10_0848__section17875624172411"><h4 class="sectiontitle">Procedure</h4><p id="cce_10_0848__p20481194104014">To use a specific NVIDIA driver version, perform the following operations to install the GPU driver of the latest version on the node:</p>
|
|
<ol id="cce_10_0848__ol105241344172213"><li id="cce_10_0848__li188216231"><span>Take the node offline and manually evict pods on the node to prevent all programs from using GPUs.</span><p><ol type="a" id="cce_10_0848__ol8571617143112"><li id="cce_10_0848__li1857161773110">Run the following command to take the node offline:<pre class="screen" id="cce_10_0848__screen44946508332">kubectl cordon <i><span class="varname" id="cce_10_0848__varname87891415201413"><NODE_NAME></span></i></pre>
|
|
<p id="cce_10_0848__p16578202263511">After the node goes offline, it will no longer support pod scheduling. You can verify this by running the following command:</p>
|
|
<pre class="screen" id="cce_10_0848__screen1517164411137">kubectl get node <i><span class="varname" id="cce_10_0848__varname20272152451420"><NODE_NAME></span></i></pre>
|
|
<p id="cce_10_0848__p131179415154">If the following information is displayed, the node does not allow pod scheduling:</p>
|
|
<pre class="screen" id="cce_10_0848__screen18500645193511">NAME STATUS ROLES AGE VERSION
|
|
192.168.1.xx Ready,<strong id="cce_10_0848__b11782144151616">SchedulingDisabled</strong> <none> 20m v1.25.5-r20-25.1.31.1</pre>
|
|
</li><li id="cce_10_0848__li16370420113317">Manually evict pods on the node.<pre class="screen" id="cce_10_0848__screen89229405372">kubectl drain 192.168.1.xx --ignore-daemonsets=true --delete-emptydir-data</pre>
|
|
<p id="cce_10_0848__p94113348376">Expected result</p>
|
|
<pre class="screen" id="cce_10_0848__screen135511598396">node/192.168.1.xx drained</pre>
|
|
</li><li id="cce_10_0848__li136511653164512">Check whether there are any DaemonSet pods are using GPUs. If so, stop kubelet and the runtime.<p id="cce_10_0848__p16741135616405"><a name="cce_10_0848__li136511653164512"></a><a name="li136511653164512"></a>Log in to the node where the GPU driver needs to be upgraded, for example, the node with IP address 192.168.1.xx.</p>
|
|
<ul id="cce_10_0848__ul93781118114112"><li id="cce_10_0848__li537811818410">Stop containerd.<pre class="screen" id="cce_10_0848__screen1395815724713">systemctl stop kubelet kubelet-monit containerd containerd-monit</pre>
|
|
</li><li id="cce_10_0848__li292103217411">Stop Docker.<pre class="screen" id="cce_10_0848__screen109901649174120">systemctl stop kubelet kubelet-monit docker docker-monit</pre>
|
|
</li></ul>
|
|
</li><li id="cce_10_0848__li14292101112318">Check if any program is using GPUs. If so, evict it.<ol class="substepthirdol" id="cce_10_0848__ol13556104025119"><li id="cce_10_0848__li13293123765114">Run the following command to check if any program is using GPUs:<pre class="screen" id="cce_10_0848__screen8756815123412">sudo fuser -v /dev/nvidia*</pre>
|
|
<div class="p" id="cce_10_0848__p14659104053114">If the following information is displayed, some programs are using the GPUs. If no fuser command is unavailable (for example, in an RPM-based Linux distribution), run the <strong id="cce_10_0848__b2190185717419">yum install psmisc</strong> command to install the Psmisc package:<pre class="screen" id="cce_10_0848__screen1672984052319"> USER PID ACCESS COMMAND
|
|
/dev/nvidia0: root 12192 F.... nvidia-gpu-devi
|
|
/dev/nvidiactl: root 12192 F.... nvidia-gpu-devi</pre>
|
|
</div>
|
|
</li><li id="cce_10_0848__li949206175218">Delete the detected process. In this example, the process ID is 12192. Replace it with the actual one.<pre class="screen" id="cce_10_0848__screen16523617522">sudo kill <i><span class="varname" id="cce_10_0848__varname1051126195211">12192</span></i></pre>
|
|
</li><li id="cce_10_0848__li1269813198523">After deleting the detected programs, run the following command again to verify:<pre class="screen" id="cce_10_0848__screen65217645211">sudo fuser -v /dev/nvidia* # Recheck if any program is using GPUs.</pre>
|
|
<p id="cce_10_0848__p115218625214">If no process is displayed in the command output, the detected processes have been evicted, and no program is using the GPUs.</p>
|
|
</li></ol>
|
|
</li></ol>
|
|
</p></li><li id="cce_10_0848__li668017530595"><span>Install the NVIDIA driver of a specified version on the node.</span><p><ol type="a" id="cce_10_0848__ol155581301619"><li id="cce_10_0848__li125584305111">Download the driver of the specified version at the <a href="https://www.nvidia.com/download/index.aspx" target="_blank" rel="noopener noreferrer">official NVIDIA website</a>. For details about how to select a proper driver version, see <a href="cce_10_0846.html">Selecting a GPU Driver Version for Nodes</a>.</li><li id="cce_10_0848__li164511013159">Record the driver information of the current version. The command for obtaining a driver varies depending on the CCE AI Suite (NVIDIA GPU) add-on version.<ul id="cce_10_0848__ul139284617488"><li id="cce_10_0848__li1139274624818">For v1.x.x, run the following command:<pre class="screen" id="cce_10_0848__screen8998123992417">/opt/cloud/cce/nvidia/bin/nvidia-smi</pre>
|
|
</li><li id="cce_10_0848__li1392134618488">For v2.0.0 through v2.5.3, run the following command:<pre class="screen" id="cce_10_0848__screen1429612819259">/usr/local/nvidia/bin/nvidia-smi</pre>
|
|
</li><li id="cce_10_0848__li5392194613481">For versions later than v2.5.4, run the following command:<pre class="screen" id="cce_10_0848__screen5374132416258">nvidia-smi</pre>
|
|
</li></ul>
|
|
<p id="cce_10_0848__p361718564231">The execution results of the preceding three versions are basically the same. The command output is as follows:</p>
|
|
<pre class="screen" id="cce_10_0848__screen06091335757">Tue Feb 20 15:06:54 2024
|
|
+-----------------------------------------------------------------------------+
|
|
| <strong id="cce_10_0848__b8411125773713">NVIDIA-SMI 470.141.03 Driver Version: 470.141.03</strong> CUDA Version: 11.4 |
|
|
|-------------------------------+----------------------+----------------------+
|
|
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
|
|
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|
|
| | | MIG M. |
|
|
|===============================+======================+======================|
|
|
| 0 Tesla V100-SXM2... Off | 00000000:21:01.0 Off | 0 |
|
|
| N/A 31C P0 23W / 300W | 0MiB / 16160MiB | 0% Default |
|
|
| | | N/A |
|
|
+-------------------------------+----------------------+----------------------+
|
|
|
|
+-----------------------------------------------------------------------------+
|
|
| Processes: |
|
|
| GPU GI CI PID Type Process name GPU Memory |
|
|
| ID ID Usage |
|
|
|=============================================================================|
|
|
| No running processes found |
|
|
+-----------------------------------------------------------------------------+</pre>
|
|
</li><li id="cce_10_0848__li19227102617215">Prepare for the installation.<div class="p" id="cce_10_0848__p27271147134510"><a name="cce_10_0848__li19227102617215"></a><a name="li19227102617215"></a>Obtain the kernel version of the current node and install its dependencies (such as gcc, make, and kernel-devel). The following is an example using a RPM-based Linux distribution (whose OS is EulerOS, HCE, or CentOS).<pre class="screen" id="cce_10_0848__screen17277478450">yum install -y perl kernel-devel-$(uname -r) zlib-devel binutils binutils-extra binutils-devel elfutils-libelf-devel gcc make</pre>
|
|
</div>
|
|
</li><li id="cce_10_0848__li47473620399">Install the downloaded driver and verify the installation.<div class="p" id="cce_10_0848__p483015416392"><a name="cce_10_0848__li47473620399"></a><a name="li47473620399"></a>The <strong id="cce_10_0848__b1334218385482">NVIDIA-Linux-x86_64-535.54.03.run</strong> driver is used as an example.<pre class="screen" id="cce_10_0848__screen13561055113913"># Obtain the kernel version.
|
|
kerVersion=`uname -r`
|
|
|
|
# Logs for installation details
|
|
touch /root/nvidia-installer.log
|
|
|
|
# Install the driver downloaded at the official NVIDIA website.
|
|
sh NVIDIA-Linux-x86_64-535.54.03.run --silent --kernel-source-path=/usr/src/kernels/$kerVersion --log-file-name=/root/nvidia-installer.log --no-install-compat32-libs --utility-prefix="/usr/local/nvidia" --opengl-prefix="/usr/local/nvidia"</pre>
|
|
</div>
|
|
<div class="note" id="cce_10_0848__note18975295338"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="cce_10_0848__p13579153912339">If the following information is displayed during the installation:</p>
|
|
<pre class="screen" id="cce_10_0848__screen1713244712334">ERROR: An NVIDIA kernel module 'nvidia' appears to already be loaded in your kernel. This may be because it is in use (for example, by an X server, a CUDA program, or the NVIDIA Persistence Daemon), but this may also happen if your kernel was configured without support for module unloading. Please be sure to exit any programs that may be using the GPU(s) before attempting to upgrade your driver. If no GPU-based programs are running, you know that your kernel supports module unloading, and you still receive this message, then an error may have occurred that has corrupted an NVIDIA kernel module's usage count, for which the <strong id="cce_10_0848__b19579739193314">simplest remedy is to reboot your computer</strong>.
|
|
ERROR: Installation has failed. Please see the file '/root/nvidia-installer.log' for details. You may find suggestions on fixing installation problems in the README available on the Linux driver download page at www.nvidia.com.</pre>
|
|
<p id="cce_10_0848__p37771601344">Perform the following operations to handle this issue:</p>
|
|
<ol id="cce_10_0848__ol14579183919336"><li id="cce_10_0848__li19655195911353">Disable kubelet and the runtime.<ul id="cce_10_0848__ul163143011373"><li id="cce_10_0848__li1363210307379">Disable containerd.<pre class="screen" id="cce_10_0848__screen12632153019372">systemctl disable kubelet kubelet-monit containerd containerd-monit</pre>
|
|
</li><li id="cce_10_0848__li1763216307378">Disable Docker.<pre class="screen" id="cce_10_0848__screen3632133014372">systemctl disable kubelet kubelet-monit docker docker-monit</pre>
|
|
</li></ul>
|
|
</li><li id="cce_10_0848__li175791539143315">Restart the affected node.</li><li id="cce_10_0848__li757913398331">Run the driver installation command again.</li><li id="cce_10_0848__li866172615343">Enable kubelet and the runtime.<ul id="cce_10_0848__ul4241128173810"><li id="cce_10_0848__li102452843812">Enable containerd.<pre class="screen" id="cce_10_0848__screen924228123813">systemctl enable kubelet kubelet-monit containerd containerd-monit</pre>
|
|
</li><li id="cce_10_0848__li17241288384">Enable Docker.<pre class="screen" id="cce_10_0848__screen62412281388">systemctl enable kubelet kubelet-monit docker docker-monit</pre>
|
|
</li></ul>
|
|
</li></ol>
|
|
</div></div>
|
|
</li><li id="cce_10_0848__li131281234143912">Check whether the new version of the GPU driver is installed.<ul id="cce_10_0848__ul1253514113515"><li id="cce_10_0848__cce_10_0848_li1139274624818">For v1.x.x, run the following command:<pre class="screen" id="cce_10_0848__cce_10_0848_screen8998123992417">/opt/cloud/cce/nvidia/bin/nvidia-smi</pre>
|
|
</li><li id="cce_10_0848__cce_10_0848_li1392134618488">For v2.0.0 through v2.5.3, run the following command:<pre class="screen" id="cce_10_0848__cce_10_0848_screen1429612819259">/usr/local/nvidia/bin/nvidia-smi</pre>
|
|
</li><li id="cce_10_0848__cce_10_0848_li5392194613481">For versions later than v2.5.4, run the following command:<pre class="screen" id="cce_10_0848__cce_10_0848_screen5374132416258">nvidia-smi</pre>
|
|
</li></ul>
|
|
<p id="cce_10_0848__p773474513296">If the following information is displayed, the new version of the GPU driver has been installed:</p>
|
|
<pre class="screen" id="cce_10_0848__screen814493420395">Tue Feb 20 15:13:58 2024
|
|
+---------------------------------------------------------------------------------------+
|
|
|<strong id="cce_10_0848__b12143234183920"> NVIDIA-SMI 535.54.03 Driver Version: 535.54.03</strong> CUDA Version: 12.2 |
|
|
|-----------------------------------------+----------------------+----------------------+
|
|
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
|
|
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
|
|
| | | MIG M. |
|
|
|=========================================+======================+======================|
|
|
| 0 Tesla V100-SXM2-16GB Off | 00000000:21:01.0 Off | 0 |
|
|
| N/A 34C P0 38W / 300W | 0MiB / 16384MiB | 2% Default |
|
|
| | | N/A |
|
|
+-----------------------------------------+----------------------+----------------------+
|
|
|
|
+---------------------------------------------------------------------------------------+
|
|
| Processes: |
|
|
| GPU GI CI PID Type Process name GPU Memory |
|
|
| ID ID Usage |
|
|
|=======================================================================================|
|
|
| No running processes found |
|
|
+---------------------------------------------------------------------------------------+</pre>
|
|
</li><li id="cce_10_0848__li1967012411454">Delete driver files and historical commands.<pre class="screen" id="cce_10_0848__screen10769193424516">rm NVIDIA-Linux-x86_64-535.54.03.run /root/nvidia-installer.log && history -c</pre>
|
|
</li></ol>
|
|
</p></li><li id="cce_10_0848__li2126119965"><span>Restart the node.</span><p><ol type="a" id="cce_10_0848__ol10939141132710"><li id="cce_10_0848__li151491011277">Restart kubelet and the runtime.<ul id="cce_10_0848__ul314940142711"><li id="cce_10_0848__li161490012276">Restart containerd.<pre class="screen" id="cce_10_0848__screen151499062718">systemctl start kubelet kubelet-monit containerd containerd-monit</pre>
|
|
</li></ul>
|
|
<ul id="cce_10_0848__ul4149190152715"><li id="cce_10_0848__li614980152715">Restart Docker.<pre class="screen" id="cce_10_0848__screen191491104278">systemctl start kubelet kubelet-monit docker docker-monit</pre>
|
|
</li></ul>
|
|
</li><li id="cce_10_0848__li46267411276">Remove scheduling restrictions on the node.<pre class="screen" id="cce_10_0848__screen31491017275">kubectl uncordon <NODE_NAME></pre>
|
|
</li></ol>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="cce_10_0845.html">GPU Driver Version</a></div>
|
|
</div>
|
|
</div>
|
|
|