Faster Downloads Using Python Multithreading

The post “Faster Downloads Using Python Multithreading” was originally published on QuantInsti blog.

Imagine you have to backtest a strategy on 50 stocks and for that you have to download price data of 50 stocks. But traditionally you have to download ticker by ticker. This sequential download process can be painfully slow, especially when each API call requires waiting for external servers to respond. What if you could download multiple stock data simultaneously?

“Multithreading does exactly that.”

In this article, we will cover the following topics:

• What is Multithreading?
• How to Implement Multithreading in Python?
• When to use and not use multithreading in Python?

Prerequisites

To fully grasp the concepts covered, it is essential to have a strong foundation in Python and financial data handling.

Start with the Python Programming blog to understand Python’s core functionalities.

Next, learn how to retrieve market data efficiently by reading Historical Market Data in Python and Basic Operations on Stock Data Using Python, which cover essential data manipulation techniques.

Additionally, Market Data FAQ provides answers to common questions about data sources, formats, and their applications in algorithmic trading.

For a structured learning approach, consider Python for Trading (Basic), a course that introduces Python essentials for trading, or Getting Market Data, which teaches how to efficiently fetch and process financial data. These resources will ensure a solid foundation before diving into more advanced topics.

Alright, let’s dive in.

What is Multithreading?

Multithreading is a programming technique that allows a program to execute multiple threads concurrently. But what exactly is a thread? A thread is the smallest sequence of instructions that can be managed independently by an operating system. You can think of a thread as a mini program running inside your main program.

When you write a Python program to download stock data, it starts with one thread called the main thread, which executes the code step by step. If you write a script to download stock prices for Apple, Microsoft, and Google, the main thread will send a request to get Apple’s stock data, wait for the response, process the data, and then move to Microsoft’s stock data, repeating the process. Since each download involves waiting for a response from the server, the program remains idle during this time. This is where multithreading comes in.

With multithreading, instead of using just the main thread, the program creates multiple threads that work simultaneously. Each thread can handle a different stock ticker, allowing downloads to happen in parallel. One thread downloads Apple’s stock data while another downloads Microsoft’s stock data, and a third handles Google’s stock data.

If one thread is waiting for a server response, the other threads continue working, reducing idle time and making the program much faster. Although it looks like all threads are running at the same time, the operating system rapidly switches between them, giving the illusion of parallel execution.

On computers with multiple processor cores, some threads can truly run in parallel, further improving performance. Because all threads share the same memory space, it is important to manage shared data properly to prevent conflicts and unexpected behavior. Now that you understand how multithreading helps speed up stock data downloads, let’s learn how to implement it in Python with a simple example.

How to Implement Multithreading in Python?

Step 1: Import the Threading Module

The first step is to import the threading module, which allows multiple functions to run concurrently.

import threading

mport_threading.py hosted with ❤ by GitHub

Step 2: Define Your Task

A function is required to define the work each thread will perform. In this example, the function simulates downloading stock data by printing a message, waiting for two seconds, and then confirming the download is complete.

import time  
def download_stock(ticker):
    print(f"Downloading {ticker}...")  
    time.sleep(2)  # Simulating a 2-second download  
    print(f"{ticker} downloaded!")  

Define_task.py hosted with ❤ by GitHub

Step 3: Create and Start Threads

Instead of running the function sequentially, separate threads are created for each task. Threads allow the tasks to start at the same time.

# Create two threads for AAPL and GOOGL  
thread1 = threading.Thread(target=download_stock, args=("AAPL",))  
thread2 = threading.Thread(target=download_stock, args=("GOOGL",))  

# Start the threads  
thread1.start()  
thread2.start()

Start_threads.py hosted with ❤ by GitHub

Step 4: Wait for Threads to Finish

To ensure all threads complete before moving forward, the .join() method is used. This prevents the program from exiting before the tasks are completed.

thread1.join()
thread2.join()
print("All downloads complete!")

Threads_to_finish.py hosted with ❤ by GitHub

Now that you have understood how to implement multithreading. Let’s dive into a practical example of downloading 5 stocks.

Practical Example: Multi-Stock Data Download

import time
import threading
import yfinance as yf  # Install with: pip install yfinance

def fetch_data(ticker):
   data = yf.download(ticker, period="1mo")
   print(f"{ticker}: {len(data)} rows")

# Without threads
start = time.time()
for ticker in ["AAPL", "GOOGL", "MSFT", "TSLA", "AMZN"]:
   fetch_data(ticker)
print(f"Sequential time: {time.time() - start:.1f}s\n")

# With threads
start = time.time()
threads = []
for ticker in ["AAPL", "GOOGL", "MSFT", "TSLA", "AMZN"]:
   thread = threading.Thread(target=fetch_data, args=(ticker,))
   thread.start()
   threads.append(thread)

for thread in threads:
   thread.join()
print(f"Threaded time: {time.time() - start:.1f}s")

Multi_stock_data_download.py hosted with ❤ by GitHub

Output: (Output times may vary depending on the device, but the threaded execution will be faster.)

AAPL: 20 rows
GOOGL: 20 rows
MSFT: 20 rows
TSLA: 20 rows
AMZN: 20 rows
Sequential time: 0.6s

AAPL: 20 rows
MSFT: 20 rows
GOOGL: 20 rows
TSLA: 20 rows
AMZN: 20 rows
Threaded time: 0.2s

As we can see, the multithreading is much faster than sequential downloads.

Important Disclaimer: While multithreading significantly speeds up data downloading, be cautious of API rate limits imposed by data providers like yfinance. Hitting these limits can lead to IP blocks or service disruptions. Always implement appropriate delays (using time.sleep()) and respect the provider’s rate limits. See yfinance documentation for reference.

When to use and not use multithreading in Python?

In this blog, we explored the implementation of multithreading and demonstrated its advantages with a practical example of downloading multiple stock tickers. It is essential to understand where multithreading works best and where it is not ideal.  Multithreading in Python is most effective when working with input and output-bound tasks where the program spends most of its time waiting for external data rather than performing computations. Input and output-bound tasks include operations such as downloading data from the internet, reading and writing files, and communicating with a database. In these cases, the CPU is often idle while waiting for responses from an external source, which allows Python to switch between multiple threads and continue executing other tasks in the meantime.

However, multithreading is not suitable for CPU-intensive tasks that involve continuous calculations, such as mathematical computations, machine learning model training, and large-scale data processing. These tasks require constant processing power, leaving no idle time for the system to efficiently switch between threads. Python’s Global Interpreter Lock, commonly known as GIL, restricts multiple threads from executing Python code in parallel within a single process. Even if multiple threads are created, only one thread executes Python code at any given time, which eliminates any real performance gain.

For CPU-intensive tasks, multiprocessing is a better alternative. Unlike threads, which share the same process, multiprocessing creates separate processes, each with its own memory space and execution environment. This allows tasks to run in parallel across multiple CPU cores, effectively bypassing the limitations imposed by the Global Interpreter Lock.

Next Steps

Once you have a strong foundation, you can explore advanced strategies and data analysis techniques. Understanding high-quality financial datasets is crucial, and the Nasdaq Data Link blog provides insights into accessing reliable market data.

Additionally, Data Preprocessing explains how to clean and refine datasets for machine learning applications, which is essential for algorithmic trading.

For hands-on experience with Python in financial markets, Python for Trading offers a deeper dive into financial data analysis and strategy development.

To ensure that trading strategies are effective, Backtesting Trading Strategies provides guidance on designing, testing, and optimizing strategies.

If you’re interested in machine learning applications in trading, Data and Feature Engineering for Trading is an excellent course that covers data transformation techniques for developing robust machine learning models.

For traders looking to enhance their knowledge in high-frequency trading (HFT) and statistical modeling, the Trading in Milliseconds by Dr. Ernest Chan course provides specialized training in order flow trading, stop hunting, spoofing, and front-running, along with Python-based implementations.

Additionally, the Advanced Algorithmic Trading Strategies learning track offers structured training in statistical analysis, machine learning, and medium-frequency trading strategies.

File in the download:

Multithreading Python Notebook

This Jupyter Notebook contains the Python implementation of all the concepts we discussed, from basic threading implementation to multi-ticker data download. Feel free to reuse the code as needed. Login to download