Using HttpClient safely in a Multi-threaded environment.

In my previous post ( Understanding commons HttpClient with example.) , we discussed about the basics of HttpClient with an example . In this post we will learn how we can use HttpClient in a multi-threaded environment.

Why use HttpClient in Multi-threaded environment ?

Now that's a good question . Suppose we need to download multiple files simultaneously .For example  in my project , we have 18 nodes on live environment . Sometimes we need to download log files from all the nodes for doing some log analysis . Now there are various options to do that . 

• The first approach is that we can download all the files one by one . Do you think that's a good approach ? I think I know your answer :)
• The second approach is we can use HttpClient with multiple threads to download all the log files concurrently . 

So what is the problem now ?

For execution , each method requires an instance of  HttpConnection.  HttpConnection can only be safely used from a single thread and method at a time . Also connections are finite resources and they should be managed in a proper way . We need  to make sure that they are properly allocated to the methods that require them . 

OK Cool . But how are we going to do that ?
MultiThreadedHttpConnectionManager to the rescue .

Below code snippet shows how we can use it with HttpClient.

MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
HttpClient client = new HttpClient(connectionManager); 

Example : 
In this example , there are three separate threads making a GET request to three different URIs . For convenience , I have taken the URIs in an array .

• Here instead of creating only HttpClient instance ( as in the previous blog ) , first an instance of MultiThreadedHttpConnectionManager   is created and then its reference is passed to the HttpClient in constructor.
• Connection is released manually in the finally block .This is because HttpClient doesn't know when a method is not using a connection as method's response is not read directly by the HttpClient. The response is read using method's connection , so a connection can not be released until the response body is read which is after the HttpClient finishes executing the method. 

package httpclient;

import java.io.IOException;

import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.MultiThreadedHttpConnectionManager;
import org.apache.commons.httpclient.StatusLine;
import org.apache.commons.httpclient.methods.GetMethod;

public class MultiThreadedExample {

    public static void main(String[] args) {

        // Creating MultiThreadedHttpConnectionManager
        MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();

        // Passing it to the HttpClient.
        HttpClient httpClient = new HttpClient(connectionManager);

        // URIs that needs to be GET by threads.
        String[] urisToGet = { "https://www.google.com/search?hl=en&q=httpclient&btnG=Google+Search",
                "https://www.google.com/search?hl=en&q=servicemix&btnG=Google+Search",
                "https://www.google.com/search?hl=en&q=tomcat&btnG=Google+Search", };

        // create a thread for each URI
        for (int i = 0; i < urisToGet.length; i++) {
            GetMethod get = new GetMethod(urisToGet[i]);
            new CreateThread(httpClient, get, "Thread-" + i).start();
        }
    }

    static class CreateThread extends Thread {
        private final HttpClient httpClient;
        private final GetMethod method;

        public CreateThread(HttpClient httpClient, GetMethod method, String name) {
            super(name);
            this.httpClient = httpClient;
            this.method = method;
        }

        @Override
        public void run() {
            try {
                System.out.println(Thread.currentThread().getName() + " - about to get something from "
                        + method.getURI());

                // Execute the method.  
                int statusCode = httpClient.executeMethod(method);

                if (statusCode != HttpStatus.SC_OK) {
                    System.err.println("Method failed: " + method.getStatusLine());
                }

                // Reading the status body.  
                StatusLine statusLine = method.getStatusLine();
                System.out.println(Thread.currentThread().getName() + " " + statusLine);
            } catch (HttpException e) {
                System.err.println("Fatal protocol violation: " + e.getMessage());
            } catch (IOException e) {
                System.err.println("Fatal transport error: " + e.getMessage());
            } finally {
                // Release the connection.  
                method.releaseConnection();
            }
        }

    }

}