Threads and Virtual Threads: Demystifying the World of Concurrency In Modern Times

Due to their ability to enable parallelism and asynchronous execution, threads have an essential role in efficiently utilizing multi-core processors. Without them, handling concurrent tasks in modern applications like real-time inference in IoT or Asynchronous I/O in AI/ML would neither be feasible nor imaginable. The arrival of virtual threads has further grown such possibilities by eliminating the sole dependency on operating system threads. Managed by the application runtime or virtual machine, virtual threads allow for more efficient concurrency management and reduced overhead associated with traditional threads. Therefore, in this blog, we will try to understand all there is to know about threads and virtual threads. Right from what threads are and where they came from up to the many advantages and disadvantages of virtual threads, we will cover everything in our discussion.

The Backbone of Parallelism

A thread is the smallest unit of processing that can be scheduled. It runs concurrently with—and largely independently of—other such units. It's an instance of java.lang.Thread, which encapsulates the execution context of a sequential flow of instructions, allowing for parallelism and asynchronous execution within a program. A Java process is made up of different threads where each thread represents a computational procedure.

There are two kinds of threads, 

1. Platform thread
2. Virtual thread

History of virtual thread

Project Loom has made it into the JDK through JEP 425. It’s available since Java 19 in September 2022 as a preview feature. Its goal is to dramatically reduce the effort of writing, maintaining, and observing high-throughput concurrent applications.

Before I go into virtual threads, I need to revisit classic threads or, what I will call them from here on out, platform threads.

The JDK implements platform threads as thin wrappers around operating system (OS) threads, which are costly, so you cannot have too many of them. In fact, the number of threads often becomes the limiting factor long before other resources, such as CPU or network connections, are exhausted.

In other words, platform threads often cap an application’s throughput to a level well below what the hardware could support. 

‍

That’s where virtual threads come in.

What is a virtual thread?

Introduced in JDK 19, Java virtual threads provide a lightweight alternative to traditional operating system (OS) threads. They aim to address the limitations of conventional thread-based concurrency models, particularly in I/O-bound scenarios. Traditional Java threads often block while waiting for I/O operations to complete, leading to inefficiencies in resource utilization and application responsiveness. 

‍

Asynchronous programming paradigms, utilizing platform threads provided by the operating system, have been adopted to mitigate these inefficiencies. However, such paradigms come with overhead in resource consumption and scalability. Virtual threads overcome these issues by allowing multiple virtual threads to be multiplexed onto a smaller number of platform threads, reducing overhead while maintaining simplicity in programming. By leveraging virtual threads, developers can efficiently manage numerous concurrent tasks, enhancing scalability, resource utilization, and responsiveness, particularly in I/O-bound applications.

‍

Virtual threads in Java offer a lightweight alternative to traditional operating system threads, managed entirely by the Java Virtual Machine (JVM). They optimize concurrency by multiplexing numerous virtual threads onto a smaller set of operating system threads, reducing overhead and simplifying concurrency programming. Particularly effective for I/O-bound tasks, virtual threads enhance application scalability and responsiveness, facilitating the development of efficient and responsive Java applications.

What is a platform thread?

Java threads are essentially wrappers over operating system threads, meaning that when a thread is created in a Java application, it requests the operating system to allocate a corresponding thread. These threads, also known as platform threads, are directly mapped to operating system threads. Each platform thread is associated with an operating system thread, and only when the platform thread terminates does the operating system thread become available for other tasks. In contrast, virtual threads are managed and scheduled by the Java Virtual Machine (JVM), offering a lightweight alternative to platform threads.

How do virtual threads work?

Virtual threads in Java are executed by the Java Virtual Machine (JVM), which manages them along with a pool of operating system (OS) threads. When a virtual thread is created and submitted for execution, the JVM determines when to map it to an available OS thread (also known as a "carrier thread") for actual execution. Virtual threads in Java, when created, enter a "ready" state in the JVM, awaiting assignment to an OS thread.

‍

• The JVM manages a pool of OS threads and assigns one as a "carrier thread" to execute the virtual thread's task.
• If a task involves blocking operations, the virtual thread can be paused without affecting the carrier thread, enabling other virtual threads to run concurrently.
• Synchronization between virtual threads is possible using traditional methods, with the JVM ensuring proper coordination.
• Upon task completion, virtual threads can be recycled for future tasks.
• If a virtual thread is paused, the JVM can switch its execution to another virtual thread or carrier thread for efficiency.
• In summary, Java's virtual threads offer lightweight and efficient concurrency, managed by the JVM through OS thread mapping and resource optimization.

Platform threads, including daemon threads, persist even after a task is completed, unlike virtual threads which are cleared once the job finishes. Virtual threads enhance efficiency by avoiding time wastage during API call waiting periods; instead of waiting, the thread is unmounted and replaced by a new virtual thread, reusing the existing platform thread. Java Virtual Threads represent a notable advancement in Java's concurrency model, offering lightweight, user-mode threading with simplicity, appealing to developers seeking efficient concurrent task management. However, virtual threads are not suitable for CPU-intensive work, and all carrier threads created for virtual threads are daemon threads, necessitating careful consideration of their pros and cons before implementation.

Virtual thread creation APIs

1. Thread.ofVirtual().start(Runnable);

Thread thread = Thread.ofVirtual().start(() -> System.out.println("Hello"));

thread.join();

‍

2. Thread.startVirtualThread() API

Runnable task = () -> { System.out.println("Hello Virtual Thread!"); };

Thread.startVirtualThread(task);

‍

3. Executors.newVirtualThreadPerTaskExecutor();

ExecutorService executorService = Executors.newVirtualThreadPerTaskExecutor();

       // Submit tasks to the executor service

executorService.submit(() -> {

   System.out.println("Task 1 is running on virtual thread: " +                      Thread.currentThread().getName());

        });

   // Shutdown the executor service

executorService.shutdown();

‍

4. Executors.newThreadPerTaskExecutor(ThreadFactory);

The method Executors.newThreadPerTaskExecutor(ThreadFactory) doesn't create virtual threads directly. Instead, it enables you to use a custom ThreadFactory to control how threads are created. To mimic virtual threads, implement the ThreadFactory interface to generate threads with names indicating their virtual nature.

public class VirtualThreadPerTaskCustomExample {

    public static void main(String[] args) {

// Create a custom thread factory for virtual threads

         ThreadFactory customThreadFactory = new VirtualThreadFactory();

// Create an executor service using the custom thread factory

         ExecutorService executorService = Executors.newThreadPerTaskExecutor(customThreadFactory);

 // Submit tasks to the executor service

        executorService.submit(() -> {

            System.out.println("Task 1 is running on virtual thread: " + Thread.currentThread().getName());

        });

        // Shutdown the executor service

executorService.shutdown();

}

   // Custom thread factory to create virtual threads

     static class VirtualThreadFactory implements ThreadFactory {

         private int count = 0;

         @Override

         public Thread newThread(Runnable r) {

             return new Thread(r, "VirtualThread-" + count++);

        }

      }

}

‍

5. Thread.ofVirtual().unstarted(Runnable);

The method Thread.ofVirtual().unstarted(Runnable) is used to create an unstarted virtual thread in Java. 

Thread virtualThread = Thread.ofVirtual().unstarted(() -> {

            System.out.println("Virtual thread is running");

        });

        // Start the virtual thread

        virtualThread.start();

        // Wait for the virtual thread to finish

        try {

            virtualThread.join();

        } catch (InterruptedException e) {

            e.printStackTrace();

        }

Concurrent URL content fetching with virtual threads

Let's consider a problem where you have a list of URLs, and you want to fetch the contents of each URL concurrently using virtual threads. Here's how you can solve it using a virtual thread:

public class VirtualThreadURLFetcher {

    public static void main(String[] args) {

        // List of URLs to fetch

        List<String> urls = new ArrayList<>();

        urls.add("https://example.com");

        urls.add("https://www.openai.com");

        urls.add("https://www.google.com");

        // Fetch contents of URLs concurrently using virtual threads

        fetchURLContentsConcurrently(urls);

    }

    // Fetch contents of URLs concurrently using virtual threads

    public static void fetchURLContentsConcurrently(List<String> urls) {

        // Create a virtual thread executor service

        ExecutorService executorService = Executors.newVirtualThreadExecutor(new VirtualThreadFactory());

        // Submit tasks to fetch URL contents

        for (String url : urls) {

            executorService.submit(() -> {

                try {

                    // Fetch URL contents

                    String content = fetchURL(url);

                    System.out.println("Fetched content from " + url + ":\n" + content);

                } catch (IOException e) {

                    e.printStackTrace();

                }

            });

        }

        // Shutdown the executor service

        executorService.shutdown();

        try {

            // Wait for all tasks to complete or timeout after 10 seconds

            executorService.awaitTermination(10, TimeUnit.SECONDS);

        } catch (InterruptedException e) {

            e.printStackTrace(); }

    }

    // Fetch content of a URL

    public static String fetchURL(String urlString) throws IOException {

        URL url = new URL(urlString);

        StringBuilder content = new StringBuilder();

        try (BufferedReader reader = new BufferedReader(new InputStreamReader(url.openStream()))) {

            String line;

            while ((line = reader.readLine()) != null) {

                content.append(line).append("\n");

            }

        }

        return content.toString();

    }

    // Custom thread factory to provide names for virtual threads

    static class VirtualThreadFactory implements ThreadFactory {

        private int count = 0;

        @Override

        public Thread newThread(Runnable r) {

            return new Thread(r, "VirtualThread-" + count++);

        }

    }

}

In this example

1. We define it in a class VirtualThreadURLFetcher.
2. Inside the main method, we create a list of URLs that we want to fetch.
3. We define a method fetchURLContentsConcurrently that takes a list of URLs and fetches their contents concurrently using virtual threads. Each URL fetch operation is represented as a task submitted to an ExecutorService.
4. Inside the fetchURLContentsConcurrently method, we submit tasks to fetch the contents of each URL to the executor service. Each task fetches the content of a URL using the fetchURL method and prints the content.
5. We define a method fetchURL that takes a URL string, opens a connection to the URL, and reads its content into a StringBuilder.
6. We define a custom ThreadFactory named VirtualThreadFactory to provide names for virtual threads.
7. Each virtual thread created by this factory will have a name in the format VirtualThread-{number}.
8. When you run this program, it will fetch the contents of each URL concurrently and print them out.

Threaded Realities of Concurrency

Despite advancements, challenges persist with both traditional threads and virtual threads due to their inherent nature of concurrency. While virtual threads offer improvements over traditional threads in terms of resource utilization, scalability, and complexity, they do not completely eliminate these challenges. Here are some of the most important ones:

• Resource Overhead: Traditional threads require significant memory and CPU resources due to their heavyweight nature.
• Limited Scalability: As the number of threads increases, overhead from thread creation and context switching can degrade performance.
• Complexity: Dealing with traditional threads involves low-level constructs like Thread objects and synchronization mechanisms, making programming error-prone.
• Concurrency Bugs: Programs using traditional threads are prone to race conditions, deadlocks, and other bugs due to managing shared resources.
• Blocking I/O: Traditional threads can waste time waiting for I/O operations to complete, leading to inefficient resource utilization.
• Context Switching Overhead: High overhead during context switching between traditional threads impacts system performance.

Virtual threads in Java offer a lightweight solution, reducing resource overhead, improving scalability, simplifying programming, enhancing I/O handling, and lowering context-switching overhead for scenarios like fetching data from multiple URLs concurrently.

Advantages of virtual threads

• Enhances application availability.
• Improves application throughput by handling more tasks concurrently.
• Helps prevent 'OutOfMemoryError: unable to create new native thread' scenarios.
• Reduces overall application memory consumption.
• Enhances code quality by simplifying concurrent programming.
• Fully compatible with Platform Threads, ensuring seamless integration.

Disadvantages of virtual threads

• Reduced suitability for CPU-intensive tasks due to multiplexing onto a smaller pool of OS threads, potentially decreasing dedicated processing power.
• Carrier thread dependencies in virtual threads can cause cascading impacts, while their reliance on the JVM makes them susceptible to its limitations or issues.
• Risk of thread starvation in scenarios with many virtual threads competing for limited carrier threads, potentially reducing overall throughput.
• Compatibility considerations when transitioning existing codebases, as some libraries or frameworks may not fully support virtual threads, leading to compatibility challenges.

Pinned virtual threads

• There are few cases where a blocking operation doesn't unmount the virtual thread from the carrier thread. As a result, the carrier thread is blocked.
• In these cases we will describe the virtual thread as pinned to the carrier thread.
• There are two cases in which a virtual thread is pinned to the carrier thread: When it executes code inside a synchronized block and when it calls a native method.

Despite advancements in concurrency abstractions such as coroutines and async/await, threads persist as essential components of concurrent programming. They underpin the seamless operation of diverse software systems in modern times, facilitating parallelism, responsiveness, and efficient resource utilization. In the world of AI/ML computations and IoT data handling, we owe it to threads and virtual threads to enable concurrent execution of tasks across multi-core processors, ensuring optimal performance and scalability.