How to use multithreading to accelerate Python code
This article mainly introduces how to use multithreading to accelerate Python code. The article is very detailed and has certain reference value. Interested friends must read it!
A lot of times, we mostly write code in Python, Python because of its simplicity, is really faster for small features, when our code performs remote requests or reads multiple files or processes certain data. In many cases, we all use a loop to read, which takes a long time to complete.
import requests
from time import time
url_list = [
"https://via.placeholder.com/400",
"https://via.placeholder.com/410",
"https://via.placeholder.com/420",
"https://via.placeholder.com/430",
"https://via.placeholder.com/440",
"https://via.placeholder.com/450",
"https://via.placeholder.com/460",
"https://via.placeholder.com/470",
"https://via.placeholder.com/480",
"https://via.placeholder.com/490",
"https://via.placeholder.com/500",
"https://via.placeholder.com/510",
"https://via.placeholder.com/520",
"https://via.placeholder.com/530",
]
def download_file(url):
html = requests.get(url, stream=True)
return html.status_code
start = time()
for url in url_list:
print(download_file(url))
print(f'Time taken: {time() - start}')
Output:
Time taken: 4.128157138824463
This is the obvious case, and this code will open each URL in turn, wait for it to load, print its status code, and then move on to the next URL. This code is very time-consuming if written as above, and is ideal for multithreading.
With multithreading, you can execute multiple tasks simultaneously with very low overhead. Let's try it next.
We implement multithreading using ThreadPoolExecutor from the current.futures library. Then let's write multithreaded code and explain how it works.
import requests
from concurrent.futures import ThreadPoolExecutor, as_completed
from time import time
url_list = [
"https://via.placeholder.com/400",
"https://via.placeholder.com/410",
"https://via.placeholder.com/420",
"https://via.placeholder.com/430",
"https://via.placeholder.com/440",
"https://via.placeholder.com/450",
"https://via.placeholder.com/460",
"https://via.placeholder.com/470",
"https://via.placeholder.com/480",
"https://via.placeholder.com/490",
"https://via.placeholder.com/500",
"https://via.placeholder.com/510",
"https://via.placeholder.com/520",
"https://via.placeholder.com/530",
]
def download_file(url):
html = requests.get(url, stream=True)
return html.status_code
start = time()
processes = []
with ThreadPoolExecutor(max_workers=10) as executor:
for url in url_list:
processes.append(executor.submit(download_file, url))
for task in as_completed(processes):
print(task.result())
print(f'Time taken: {time() - start}')
Output:
Time taken: 0.4583399295806885
Code processing speed increased 9 times! If there are more URLs, the performance difference should be more pronounced.
Why multithreading is so fast. When we call executor.submit, we add a new task to the thread pool.
Then what the hell is going on? When called, executor.submit We are adding a new task to the thread pool. Link it up and store it, after which we will facilitate the task call and print the results.
The as_completed method is used to pull a task from the task list to execute immediately after a task is completed. Tasks are marked as complete only if they have been executed or cancelled. We can also pass a timeout parameter to it, and if the task takes longer than this time period, even as_completed switches the task.
That's all for "How to accelerate Python code with multithreading", thanks for reading! Hope to share the content to help everyone, more relevant knowledge, welcome to pay attention to the industry information channel!
