My environment:
export ONEAPI_DEVICE_SELECTOR="level_zero:gpu"
$ sycl
Warning: ONEAPI_DEVICE_SELECTOR environment variable is set to level_zero:gpu.
To see the correct device id, please unset ONEAPI_DEVICE_SELECTOR.
[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]
[ext_oneapi_level_zero:gpu:1] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]
[ext_oneapi_level_zero:gpu:2] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]
[ext_oneapi_level_zero:gpu:3] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]
[ext_oneapi_level_zero:gpu:4] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]
[ext_oneapi_level_zero:gpu:5] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26241]
And I have following codes in file test_multi_process.py:
import os
from multiprocessing import Pool
import time
import dpctl
env_var = "ONEAPI_DEVICE_SELECTOR"
backend = "level_zero"
device_type = "gpu"
def get_dev():
dev_ids = []
for dev in dpctl.get_devices(backend=backend, device_type=device_type):
# device filter_string with format: "backend:device_type:relative_id"
# dev_ids.append(int(dev.filter_string.split(":")[-1]))
dev_ids.append(dev.filter_string)
return dev_ids
def set_env_var(dev_id):
env_val = f"{backend}:{dev_id}"
print(f"[func] [{os.getpid()}] set {env_var} = {env_val}")
os.environ[env_var] = env_val
def func(x):
print(f"[func] [{os.getpid()}] >>>>>>>>>>>>>>>>>>>>>>")
set_env_var(x)
env_val = os.environ.get(env_var, None)
print(f"[func] [{os.getpid()}] x = {x}")
print(f"[func] [{os.getpid()}] {env_var} = {env_val}")
dev_ids = get_dev()
print(f"[func] [{os.getpid()}] dev_ids = {dev_ids}")
print(f"[func] [{os.getpid()}] <<<<<<<<<<<<<<<<<<<<<<\n")
def main():
env_val = os.environ.get(env_var, None)
print(f"[main] [{os.getpid()}] {env_var} = {env_val}")
dev_ids = get_dev()
print(f"[main] [{os.getpid()}] dev_ids = {dev_ids}")
with Pool(5) as p:
p.map(func, [1, 2, 3, 4, 5])
if __name__ == '__main__':
main()Running test_multi_process.py
python test_multi_process.py
With following logs:
[main] [122421] ONEAPI_DEVICE_SELECTOR = level_zero:gpu
[main] [122421] dev_ids = ['level_zero:gpu:0', 'level_zero:gpu:1', 'level_zero:gpu:2', 'level_zero:gpu:3', 'level_zero:gpu:4', 'level_zero:gpu:5']
# sub process 1
[func] [122440] >>>>>>>>>>>>>>>>>>>>>>
[func] [122440] set ONEAPI_DEVICE_SELECTOR = level_zero:1
[func] [122440] x = 1
[func] [122440] ONEAPI_DEVICE_SELECTOR = level_zero:1
[func] [122440] dev_ids = ['level_zero:gpu:0', 'level_zero:gpu:1', 'level_zero:gpu:2', 'level_zero:gpu:3', 'level_zero:gpu:4', 'level_zero:gpu:5']
[func] [122440] <<<<<<<<<<<<<<<<<<<<<<
# sub process 2
[func] [122441] >>>>>>>>>>>>>>>>>>>>>>
[func] [122441] set ONEAPI_DEVICE_SELECTOR = level_zero:2
[func] [122441] x = 2
[func] [122441] ONEAPI_DEVICE_SELECTOR = level_zero:2
[func] [122441] dev_ids = ['level_zero:gpu:0', 'level_zero:gpu:1', 'level_zero:gpu:2', 'level_zero:gpu:3', 'level_zero:gpu:4', 'level_zero:gpu:5']
[func] [122441] <<<<<<<<<<<<<<<<<<<<<<
# sub process 3
[func] [122442] >>>>>>>>>>>>>>>>>>>>>>
[func] [122442] set ONEAPI_DEVICE_SELECTOR = level_zero:3
[func] [122442] x = 3
[func] [122442] ONEAPI_DEVICE_SELECTOR = level_zero:3
[func] [122442] dev_ids = ['level_zero:gpu:0', 'level_zero:gpu:1', 'level_zero:gpu:2', 'level_zero:gpu:3', 'level_zero:gpu:4', 'level_zero:gpu:5']
[func] [122442] <<<<<<<<<<<<<<<<<<<<<<
# sub process 4
[func] [122443] >>>>>>>>>>>>>>>>>>>>>>
[func] [122443] set ONEAPI_DEVICE_SELECTOR = level_zero:4
[func] [122443] x = 4
[func] [122443] ONEAPI_DEVICE_SELECTOR = level_zero:4
[func] [122443] dev_ids = ['level_zero:gpu:0', 'level_zero:gpu:1', 'level_zero:gpu:2', 'level_zero:gpu:3', 'level_zero:gpu:4', 'level_zero:gpu:5']
[func] [122443] <<<<<<<<<<<<<<<<<<<<<<
# sub process 5
[func] [122444] >>>>>>>>>>>>>>>>>>>>>>
[func] [122444] set ONEAPI_DEVICE_SELECTOR = level_zero:5
[func] [122444] x = 5
[func] [122444] ONEAPI_DEVICE_SELECTOR = level_zero:5
[func] [122444] dev_ids = ['level_zero:gpu:0', 'level_zero:gpu:1', 'level_zero:gpu:2', 'level_zero:gpu:3', 'level_zero:gpu:4', 'level_zero:gpu:5']
[func] [122444] <<<<<<<<<<<<<<<<<<<<<<
In the main process, I want to use GPU ID [1,2,3,4,5], and create 5 process to use one of the GPUs in each process.
My question is that:
I have set the environment variable ONEAPI_DEVICE_SELECTOR in each sub-process with using only one GPU, but the sub-process could still can see the 6 GPUs.
Is that need reload the dpcpp in python codes?
Or that ONEAPI_DEVICE_SELECTOR with dpctl can't be used in nested case?
Or that ONEAPI_DEVICE_SELECTOR should work with command syc-ls?
My environment:
And I have following codes in file
test_multi_process.py:Running
test_multi_process.pyWith following logs:
In the main process, I want to use
GPU ID [1,2,3,4,5], and create 5 process to use one of the GPUs in each process.My question is that:
I have set the environment variable
ONEAPI_DEVICE_SELECTORin each sub-process with using only one GPU, but the sub-process could still can see the 6 GPUs.Is that need reload the dpcpp in python codes?
Or that
ONEAPI_DEVICE_SELECTORwithdpctlcan't be used in nested case?Or that
ONEAPI_DEVICE_SELECTORshould work with commandsyc-ls?