******** pre-installation actions
******** install CUDA
// choose architecture, os, and etc.
// choose deb(local).
// installation instructions appear.
$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
$ sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
$ wget http://developer.download.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda-repo-ubuntu1804-10-1-local-10.1.243-418.87.00_1.0-1_amd64.deb
$ sudo dpkg -i cuda-repo-ubuntu1804-10-1-local-10.1.243-418.87.00_1.0-1_amd64.deb
$ sudo apt-key add /var/cuda-repo-10-1-local-10.1.243-418.87.00/7fa2af80.pub
$ sudo apt-get update
$ sudo apt-get -y install cuda
******** post-installation actions
**** PATH:
// The nsight-compute directory path in the above link is incorrect.
$ vi ~/.profile
# set PATH so it includes nsight-compute (co-installed with CUDA)
if [ -d "/opt/nvidia/nsight-compute/2019.4.0" ] ; then
PATH="/opt/nvidia/nsight-compute/2019.4.0:$PATH"
fi
# set PATH so it includes CUDA
if [ -d "/usr/local/cuda-10.1/bin" ] ; then
PATH="/usr/local/cuda-10.1/bin:$PATH"
fi
**** persistence daemon:
// It reduces the startup time of GPU. GPU is in lower-power state when it is not used. It takes time to start up.
// I don't need to turn in on.
**** verify:
$ source ~/.profile
$ cuda-install-samples-10.1.sh ./cuda
$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 430.50 Thu Sep 5 22:36:31 CDT 2019
GCC version: gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
$ cd cuda/NVIDIA_CUDA-10.1_Samples
$ make
$ ./bin/x86_64/linux/release/deviceQuery
./bin/x86_64/linux/release/deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
cudaGetDeviceCount returned 803
-> system has unsupported display driver / cuda driver combination
Result = FAIL
$ apt list --installed |grep nvidia
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
libnvidia-cfg1-418/unknown,now 418.87.00-0ubuntu1 amd64 [installed,automatic]
libnvidia-common-418/unknown,now 418.87.00-0ubuntu1 all [installed,automatic]
libnvidia-common-430/bionic,bionic,now 430.50-0ubuntu0.18.04.1 all [installed,auto-removable]
libnvidia-compute-418/unknown,now 418.87.00-0ubuntu1 amd64 [installed,automatic]
libnvidia-decode-418/unknown,now 418.87.00-0ubuntu1 amd64 [installed,automatic]
libnvidia-encode-418/unknown,now 418.87.00-0ubuntu1 amd64 [installed,automatic]
libnvidia-fbc1-418/unknown,now 418.87.00-0ubuntu1 amd64 [installed,automatic]
libnvidia-gl-418/unknown,now 418.87.00-0ubuntu1 amd64 [installed,automatic]
libnvidia-ifr1-418/unknown,now 418.87.00-0ubuntu1 amd64 [installed,automatic]
nvidia-compute-utils-418/unknown,now 418.87.00-0ubuntu1 amd64 [installed,automatic]
nvidia-dkms-418/unknown,now 418.87.00-0ubuntu1 amd64 [installed,automatic]
nvidia-driver-418/unknown,now 418.87.00-0ubuntu1 amd64 [installed,automatic]
nvidia-kernel-common-418/unknown,now 418.87.00-0ubuntu1 amd64 [installed,automatic]
nvidia-kernel-source-418/unknown,now 418.87.00-0ubuntu1 amd64 [installed,automatic]
nvidia-modprobe/unknown,now 418.87.00-0ubuntu1 amd64 [installed,automatic]
nvidia-prime/bionic-updates,bionic-updates,now 0.8.8.2 all [installed,automatic]
nvidia-settings/unknown,now 418.87.00-0ubuntu1 amd64 [installed,automatic]
nvidia-utils-418/unknown,now 418.87.00-0ubuntu1 amd64 [installed,automatic]
xserver-xorg-video-nvidia-418/unknown,now 418.87.00-0ubuntu1 amd64 [installed,automatic]
// Check drivers GeForce RTX 2070 in https://www.geforce.com/drivers and find driver 418, which does not support 2070 Super.
$ sudo apt purge nvidia*
$ sudo apt autoremove
// at this time, download deb(network) and install with network, since I guess the network install might have more recent versions.
$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
$ sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
$ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
$ sudo add-apt-repository "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"
$ sudo apt update
$ sudo apt install cuda
...
0 upgraded, 267 newly installed, 0 to remove and 60 not upgraded.
Need to get 1866 MB/1913 MB of archives.
After this operation, 4817 MB of additional disk space will be used.
// and reboot
$ sudo reboot now
~/works/cuda/NVIDIA_CUDA-10.1_Samples$ ./bin/x86_64/linux/release/deviceQuery
./bin/x86_64/linux/release/deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "GeForce RTX 2070 SUPER"
CUDA Driver Version / Runtime Version 10.1 / 10.1
CUDA Capability Major/Minor version number: 7.5
Total amount of global memory: 7952 MBytes (8338604032 bytes)
(40) Multiprocessors, ( 64) CUDA Cores/MP: 2560 CUDA Cores
GPU Max Clock rate: 1815 MHz (1.81 GHz)
Memory Clock rate: 7001 Mhz
Memory Bus Width: 256-bit
L2 Cache Size: 4194304 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 1024
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 3 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device supports Compute Preemption: Yes
Supports Cooperative Kernel Launch: Yes
Supports MultiDevice Co-op Kernel Launch: Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 8 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.1, CUDA Runtime Version = 10.1, NumDevs = 1
Result = PASS
// I don't know if PASS is due to network install, or reboot.
// "apt list --installed |grep nvidia" still shows nvidia-driver-418
// before reboot, the following version was "430.50" not "418.87". Maybe this was the reason of failure.
$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 418.87.01 Wed Sep 25 06:00:38 UTC 2019
GCC version: gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
$ ./bin/x86_64/linux/release/bandwidthTest
[CUDA Bandwidth Test] - Starting...
Running on...
Device 0: GeForce RTX 2070 SUPER
Quick Mode
Host to Device Bandwidth, 1 Device(s)
PINNED Memory Transfers
Transfer Size (Bytes) Bandwidth(GB/s)
32000000 13.2
Device to Host Bandwidth, 1 Device(s)
PINNED Memory Transfers
Transfer Size (Bytes) Bandwidth(GB/s)
32000000 13.5
Device to Device Bandwidth, 1 Device(s)
PINNED Memory Transfers
Transfer Size (Bytes) Bandwidth(GB/s)
32000000 381.0
Result = PASS
NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
// But I found that CUDA toolkit is not needed for docker container. Only cuda-drivers package is needed.
$ sudo apt purge nvidia*
$ sudo apt autoremove
$ sudo apt install cuda-drivers
...
0 upgraded, 146 newly installed, 0 to remove and 60 not upgraded.
Need to get 113 MB/113 MB of archives.
After this operation, 644 MB of additional disk space will be used.
----------------
2019.10.14
during tensorflow run in a docker, an error:
RuntimeError: CUDA runtime API error cudaErrorInsufficientDriver (35):
CUDA driver version is insufficient for CUDA runtime version
But the latest cuda-driver is 418.87 currently.
And I found cuda-driver is a part of nvidia-driver.
So I try to remove cuda-driver, and install nvidia-driver-435, the latest now.
$ sudo apt purge nvidia*
// this removes cuda-driver, too
$ sudo apt autoremove
$ sudo apt install nvidia-driver-435
$ sudo reboot now
// needed ?
$ ./works/cuda/NVIDIA_CUDA-10.1_Samples/1_Utilities/deviceQuery/deviceQuery
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "GeForce RTX 2070 SUPER"
CUDA Driver Version / Runtime Version 10.1 / 10.1
CUDA Capability Major/Minor version number: 7.5
Total amount of global memory: 7982 MBytes (8370061312 bytes)
(40) Multiprocessors, ( 64) CUDA Cores/MP: 2560 CUDA Cores
GPU Max Clock rate: 1815 MHz (1.81 GHz)
Memory Clock rate: 7001 Mhz
Memory Bus Width: 256-bit
L2 Cache Size: 4194304 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 1024
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 3 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device supports Compute Preemption: Yes
Supports Cooperative Kernel Launch: Yes
Supports MultiDevice Co-op Kernel Launch: Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 8 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.1, CUDA Runtime Version = 10.1, NumDevs = 1
Result = PASS
$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 435.21 Sun Aug 25 08:17:57 CDT 2019
GCC version: gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
// nvidia-cotainer-tookit was removed by "purge nvidia*"
$ sudo apt update
$ sudo apt install nvidia-container-toolkit
$ sudo systemctl restart docker