plex 參考來源
https://jocke.no/2022/02/23/plex-gpu-transcoding-in-docker-on-lxc-on-proxmox/
Pve host
特權 Lxc功能:fuse=1,nesting=1
主機和 LXC 容器都需要運行相同的驅動程式版本
1
2
3
4
5
6
7
8
9
10
11
12
13
|
#驅動黑名單:開機不載入
echo -e "blacklist nouveau\noptions nouveau modeset=0" > /etc/modprobe.d/blacklist-nouveau.conf
nano /etc/kernel/cmdline
add
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on
#更新開機模組
update-initramfs -u
proxmox-boot-tool refresh
#安裝無頭驅動
apt install pve-headers-$(uname -r)
#nvidia drive
wget -O NVIDIA-Linux-x86_64-510.47.03.run https://us.download.nvidia.com/XFree86/Linux-x86_64/510.47.03/NVIDIA-Linux-x86_64-510.47.03.run
chmod +x NVIDIA-Linux-x86_64-510.47.03.run
|
answer “no” when it asks if you want to install 32bit compability drivers
回答“否”當它詢問您是否要安裝 32 位元相容驅動程式時
answer “no” when it asks if it should update X config
回答“否”當它詢問是否應該更新 X-windows 配置時
1
|
./NVIDIA-Linux-x86_64-510.47.03.run
|
新增一些 udev 規則。這是為了確保載入正確的核心模組，並在啟動時建立所有相關的設備檔案
1
|
echo -e '\n# load nvidia modules\nnvidia-drm\nnvidia-uvm' >> /etc/modules-load.d/modules.conf
|
將以下內容添加到 /etc/udev/rules.d/70-nvidia.rules
啟動時會在/dev/中建立相關的裝置文件
1
|
nano /etc/udev/rules.d/70-nvidia.rules
|
添加以下
1
2
3
|
KERNEL=="nvidia", RUN+="/bin/bash -c '/usr/bin/nvidia-smi -L && /bin/chmod 666 /dev/nvidia*'"
KERNEL=="nvidia_uvm", RUN+="/bin/bash -c '/usr/bin/nvidia-modprobe -c0 -u && /bin/chmod 0666 /dev/nvidia-uvm*'"
SUBSYSTEM=="module", ACTION=="add", DEVPATH=="/module/nvidia", RUN+="/usr/bin/nvidia-modprobe -m"
|
避免在不使用 GPU 時卸載驅動程式/核心模組，應該執行 Nvidia 提供的持久性服務,驅動程式安裝後就可以使用它
https://docs.nvidia.com/deploy/driver-persistence/index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
# 複製並解壓縮
cp /usr/share/doc/NVIDIA_GLX-1.0/samples/nvidia-persistenced-init.tar.bz2 .
bunzip2 nvidia-persistenced-init.tar.bz2
tar -xf nvidia-persistenced-init.tar
# 刪除舊的（如果有的話,以避免屏蔽服務）
rm /etc/systemd/system/nvidia-persistenced.service
# 安裝
chmod +x nvidia-persistenced-init/install.sh
./nvidia-persistenced-init/install.sh
# 檢查是否正常
systemctl status nvidia-persistenced.service
rm -rf nvidia-persistenced-init*
|
如果沒有任何錯誤，就可以重新啟動 Proxmox 主機。重新啟動後，應該會看到以下輸出
示例：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.157 Driver Version: 390.157 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 650 Off | 00000000:05:00.0 N/A | N/A |
| 27% 42C P8 N/A / N/A | 183MiB / 978MiB | N/A Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 Not Supported |
+-----------------------------------------------------------------------------+
|
添加 Lxc conf 內容
注意Pvehost group ID,每台主機不一,需修正
Pve host:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
ls -alh /dev/nvidia*
crw-rw-rw- 1 root root 195, 0 12月 15 07:01 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 12月 15 07:01 /dev/nvidiactl
crw-rw-rw- 1 root root 195, 254 12月 15 07:02 /dev/nvidia-modeset
crw-rw-rw- 1 root root 510, 0 12月 15 07:01 /dev/nvidia-uvm
crw-rw-rw- 1 root root 510, 1 12月 15 07:01 /dev/nvidia-uvm-tools
/dev/nvidia-caps:
總用量 0
drw-rw-rw- 2 root root 80 12月 15 07:01 .
drwxr-xr-x 23 root root 5.6K 12月 15 11:20 ..
cr-------- 1 root root 236, 1 12月 15 07:01 nvidia-cap1
cr--r--r-- 1 root root 236, 2 12月 15 07:01 nvidia-cap2
lxc conf add
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 510:* rwm
lxc.cgroup2.devices.allow: c 236:* rwm
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-caps/nvidia-cap1 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-caps/nvidia-cap2 none bind,optional,create=file
|
Lxc 容器內操作
Lxc開機,進入Lxc執行以下
1
2
3
4
5
6
7
|
wget -ONVIDIA-Linux-x86_64-510.47.03.run https://us.download.nvidia.com/XFree86/Linux-x86_64/510.47.03/NVIDIA-Linux-x86_64-510.47.03.run
chmod +x NVIDIA-Linux-x86_64-510.47.03.run
./NVIDIA-Linux-x86_64-510.47.03.run --check
# answer "no" when it asks if it should update X config
./NVIDIA-Linux-x86_64-510.47.03.run --no-kernel-module
#安裝完驅動,驗證Lxc nvidia-smi輸出,如錯誤！檢查驅動版本與host一致否與 lxc.conf
nvidia-smi
|
安裝 docker 相關
1
2
3
4
5
6
7
8
9
10
11
12
|
#刪除使用apt安裝之docker
apt remove docker-compose docker docker.io containerd runc
# 從官方儲存庫安裝 docker
apt update
apt install ca-certificates curl gnupg lsb-release
curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
$(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt update
apt install docker-ce docker-ce-cli containerd.io
|
安裝 docker-compose
1
2
|
curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
|
docker-compose bash 補全
1
2
3
|
curl \
-L https://raw.githubusercontent.com/docker/compose/1.29.2/contrib/completion/bash/docker-compose \
-o /etc/bash_completion.d/docker-compose
|
安裝 nvidia-docker2
新版本的工具包需要不同的儲存庫
1
2
3
4
5
6
7
8
9
10
|
apt install -y curl
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
keyring_file="/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg"
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o ${keyring_file}
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed "s#deb https://#deb [signed-by=${keyring_file}] https://#g" | \
tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt update
apt install nvidia-docker2
|
重新啟動systemd + docker（不重新載入systemd，docker 可能無法運作）
1
2
|
systemctl daemon-reload
systemctl restart docker
|
Plex docker compose
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
version: '3.7'
services:
plex:
container_name: plex
hostname: plex
image: linuxserver/plex:latest
restart: unless-stopped
deploy:
resources:
reservations:
devices:
- capabilities: [gpu]
environment:
TZ: Europe/Paris
PUID: 0
PGID: 0
VERSION: latest
NVIDIA_VISIBLE_DEVICES: all
NVIDIA_DRIVER_CAPABILITIES: compute,video,utility
network_mode: host
volumes:
- /srv/config/plex:/config
- /storage/media:/data/media
- /storage/temp/plex/transcode:/transcode
- /storage/temp/plex/tmp:/tmp
|
內核有更新時需
卸載nvidia-drive重新安裝pve-headers,重開後重安裝nvidia驅動與pve-headers
1
2
|
./NVIDIA-Linux-x86_64-510.47.03.run --uninstall
apt install pve-headers-$(uname -r)
|