之前使用 nvidia-smi ，只能显出服务器的资源使用情况，无法看到是谁在用。
发现gpustat还不错，首先要安装一下：
pip install gpustat
使用：
gpustat
Options:
--color : Force colored output (even when stdout is not a tty)
--no-color : Suppress colored output
-u, --show-user : Display username of the process owner
-c, --show-cmd : Display the process name
-p, --show-pid : Display PID of the process
-P, --show-power : Display GPU power usage and/or limit (draw or draw,limit)
--watch, -i, --interval : Run in watch mode (equivalent to watch gpustat) if given. Denotes interval between updates. (#41)
--json : JSON Output (Experimental, #10)
Tips
To periodically watch, try gpustat --watch or gpustat -i (#41).
For older versions, one may use watch --color -n1.0 gpustat --color。
Running nvidia-smi daemon (root privilege required) will make the query much faster and use less CPU (#54).
The GPU ID (index) shown by gpustat (and nvidia-smi) is PCI BUS ID, while CUDA differently assigns the fastest GPU with the lowest ID by default. Therefore, in order to make CUDA and gpustat use same GPU index, configure the CUDA_DEVICE_ORDER environment variable to PCI_BUS_ID (before setting CUDA_VISIBLE_DEVICES for your CUDA program):
export CUDA_DEVICE_ORDER=PCI_BUS_ID.
---------------------
--no-color : Suppress colored output
-u, --show-user : Display username of the process owner
-c, --show-cmd : Display the process name
-p, --show-pid : Display PID of the process
-P, --show-power : Display GPU power usage and/or limit (draw or draw,limit)
--watch, -i, --interval : Run in watch mode (equivalent to watch gpustat) if given. Denotes interval between updates. (#41)
--json : JSON Output (Experimental, #10)
Tips
To periodically watch, try gpustat --watch or gpustat -i (#41).
For older versions, one may use watch --color -n1.0 gpustat --color。
Running nvidia-smi daemon (root privilege required) will make the query much faster and use less CPU (#54).
The GPU ID (index) shown by gpustat (and nvidia-smi) is PCI BUS ID, while CUDA differently assigns the fastest GPU with the lowest ID by default. Therefore, in order to make CUDA and gpustat use same GPU index, configure the CUDA_DEVICE_ORDER environment variable to PCI_BUS_ID (before setting CUDA_VISIBLE_DEVICES for your CUDA program):
export CUDA_DEVICE_ORDER=PCI_BUS_ID.
---------------------
