|
首先确保安装了对应的tqdmpython库
pip install tqdm
方法1：使用p_tqdm进行并行tqdm操作
对应的p_tqdm教学链接如下：
from p_tqdm import p_map
def add(a,b):
return a+b
added = p_map(add,['1','2','3'],['a','b','c'])
这里的add函数对于数组之中的每一个数值进行操作，此时放入两个对应的数组时可以进行并行操作  从中可以看出如果有多个输出内容的话，会以一种dict字典的形式返回出来，此时如果需要多个数组接收的话 完整代码如下：
from functools import partial
from p_tqdm import p_map
l1 = ['1', '2', '3']
l2 = ['a', 'b', 'c']
def add(a, b, c=''):
return a+b+c,a+b
added = p_map(partial(add, c='!'), l1, l2)
# added == ['1a!', '2b!', '3c!']
print('added = ')
print(added)
def get0(added):
return added[0]
def get1(added):
return added[1]
added0 = p_map(get0,added)
added1 = p_map(get1,added)
print('added0 = ')
print(added0)
print('added1 = ')
print(added1)
方法2：动态调用tqdm
pbar = tqdm(total = len(load_dict))
results = Parallel(n_jobs=num_jobs, backend="multiprocessing")(
delayed(get_feat_helper)(load_dict,ids) for ids in ids_splits
)
然后并行调用的时候
def get_feat_helper(load_dict,ids):
global pbar
for index in ids:
......
pbar.update(1)
|