How to Parallelize a for Loop in Python

0 min read 104 words

If you need to run a for loop in parallel, then you can do one of the following:

Option 1 – Using multiprocessing

import multiprocessing

def sumall(value):
    return sum(range(1, value + 1))

pool_obj = multiprocessing.Pool()

answer = pool_obj.map(sumall,range(0,5))
print(answer)

Option 2 – Using joblib module

from joblib import Parallel, delayed
import math

def sqrt_func(i, j):
    time.sleep(1)
    return math.sqrt(i**j)

Parallel(n_jobs=2)(delayed(sqrt_func)(i, j) for i in range(5) for j in range(2))

Option 3 – Using asyncio

import asyncio
import time

def background(f):
    def wrapped(*args, **kwargs):
        return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs)
    return wrapped

@background
def your_function(argument):
    time.sleep(2)
    print('function finished for '+str(argument))

for i in range(10):
    your_function(i)

print('loop finished')
Tags:
Andrew
Andrew

Andrew is a visionary software engineer and DevOps expert with a proven track record of delivering cutting-edge solutions that drive innovation at Ataiva.com. As a leader on numerous high-profile projects, Andrew brings his exceptional technical expertise and collaborative leadership skills to the table, fostering a culture of agility and excellence within the team. With a passion for architecting scalable systems, automating workflows, and empowering teams, Andrew is a sought-after authority in the field of software development and DevOps.

Tags