Welcome!

Machine Learning Authors: Pat Romanski, Elizabeth White, Yeshim Deniz, Corey Roth, Liz McMillan

Related Topics: Java IoT, Open Source Cloud, Machine Learning

Java IoT: Blog Post

Long Polling Explained

How long polling could be used in Spring MVC web applications

Let us start with the news publishing service called NewsPostingThread, which produces news messages like: "broadcasting news article # 1". In the heart of this service is lockObject which serves as a synchronization locking mechanism. When a thread processing request asks for latest news, it blocks by the lockObject synchronization lock until the thread generating the news will call lockObject.notifyAll(). At that point the thread processing request will prepare and send response to the client.

@Component
public class NewsPostingThread implements Runnable {
...
@Override
public void run() {
while (true) {
try {
Thread.sleep(random.nextInt(10000));
} catch (Throwable t) {
t.printStackTrace(); }
counter.incrementAndGet();
if (counter.get() > Long.MAX_VALUE - 10) {
counter.set(0);
}
synchronized(lockObject) {
lockObject.notifyAll();
}
}
}
public String getLatestNews() {
try {
log.debug("waiting for the latest news");
synchronized(lockObject) {
lockObject.wait();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
return "broadcastig news article # " + counter.get();
}  }

Let us now take a look at the part of the code that of the Spring MVC controller that calls NewsPostingThread to get the news:

@Controller
public class LongWaitController {
...
@RequestMapping(value="/longPollingAjax")
@ResponseBody
public String ajaxReply(final HttpServletRequest request,
final HttpServletResponse response,
ModelMap mm) throws Exception {
log.debug("ajaxReply");
String news = newsPostingThread.getLatestNews();
log.debug("Data is " + news);
return "{\"data\": \"" + news + "\" }";
}
}

ajaxReply(...) is a simple controller method implementation that processes long polling AJAX requests and returns raw data to the calling client. When processing the client request, a method called getLatestNews() gets blocked until the news become available. Now let us look at the client side where long polling request originates. Ajax is used to create request, and there are two ways AJAX could be used to implement long polling. One way is to use setInterval JavaScript function as a mechanism to schedule polling: it will call poll() method repeatedly at specified intervals. You might need to coordinate request timeout setting on the server and the interval you used in the setInterval functions in such a way that your requests don't start queuing up, as AJAX requests that might not return in the same order they were issued. In the code below, once user clicks on the button with id send, the poll function will be invoked

$("#send").unbind("click").bind("click", function(event) {
$("#result").html('Started looking for news ....');
/* Send the data using post and put the results in a div */
setInterval(function() { poll(); }, 200);
});
function poll() {
$.ajax({
url: "/app/longPollingAjax",
type: "GET",
dataType: "json",
success: function(response){
$("#result").html('Submitted successfully ' + response.data);
},
error:function(){
$("#result").html('There is error while submit');
}
});
}

Alternatively, a better solution is to use JQuery Ajax complete method, which fires when the Ajax call is complete after the success and error call backs have been executed. This way, when button with id send is pushed, the first poll() request is initiated. If request is successful, then the result div will show the latest news from the server, after which "complete" is called and a new poll request is issued.  If request times out or errors out, again "complete" is called and a new poll request is issued.

$("#send").unbind("click").bind("click", function(event) {
$("#result").html('Started looking for news ....');
/* Send the data using post and put the results in a div */
poll();
});
function poll() {
$.ajax({
url: "/app/longPollingAjax",
type: "GET",
dataType: "json",
success: function(response){
$("#result").html('Submitted successfully ' + response.data);
},
error:function(){
$("#result").html('There is error while submit');
},
complete: function() {
poll(); //will re-issue polling request after current request completes.
}
});
}

Enter Servlet spec 3.0

Before Servlet 3.0 spec, there were two server threading models: thread per connection and thread per request. In the thread per connection model, the thread is associated for every TCP/IP connection, and the server can scale to very high number of requests per second when they come from the same clients. Yet this model, for a non-long-polling site can have a hard time scaling to support thousands of users. The reason for it, is that for the most web sites, the users initiate an action and then the connection stays mostly idle while users read the pages and decide on the next actions. Hence threads that are tied to a connection are sitting idle. To improve scalability, the web-servers have a thread per request model. In this model, after servicing the request, the thread can be reused to service a request from different client. This model allows much greater scaling of user base at a minor expense of increased time of servicing each request. This expense is due to the thread scheduling that takes place.

Yet, with the long polling, the difference in scalability between request per connection and thread per request is blurred. It happens because each request is stuck waiting for an event (in this case news event) before generating the response. Waiting in the servlet is an inefficient, since, at the very least, the server thread that could be used to serve a different request is blocked. This results in poor scalability as users are added to the application.

Servlet 3.0 spec comes to the rescue - it defines an asynchronous way for server to process requests. To do that, servlet passes request to an AsyncContext. Then, the response processing is passed to a different thread, and the servlet processing thread can handle a request from a different user. Ideally, the web server container would maintain at least two thread pools - one to process servlet requests and another one that is used to process long running request that are passed to asynchronous context.

Let us see how this long polling example would look like if asynchronous features of Servlet 3.0 container using Spring MVC asynchronous servlet abstractions were to be used to solve the previous task.

@Controller
public class SubscribeToBroadcast {
...
@RequestMapping(value="/pollBroadcast")
@ResponseBody
public DeferredResult<String> ajaxReply(final HttpServletRequest request,
final HttpServletResponse response,
ModelMap mm) throws Exception {
final DeferredResult<String> dr = new DeferredResult<String>(
TimeUnit.MINUTES.toMillis(1), TIMEOUT_RESULT);
broadcastCounter.addSubscribed(dr);
return dr;
}
}

AjaxReply method here returns a DefferedResult, which allows the result to be processed in a thread of developers' choice. In this case the method returns a string representation of a Jason object, hence DefferedResult<String> is used. A new differed result is created that will expire in 1 minute with a default response upon expiry. Then the deferred result is added to the news subscription list and the servlet thread is done.

Let us take a look at the BroadcastCounter - the news publishing service.

@Component
public class BroadcastCounter {
private static final Logger log = Logger.getLogger(BroadcastCounter.class);
private Thread t;
private AtomicLong counter = new AtomicLong();
private List<DeferredResult<String>> subscribedClient = Collections.synchronizedList(new ArrayList<DeferredResult<String>>());

public BroadcastCounter() {
t = new Thread(
new Runnable() {
@Override
public void run() {
while(true) {
counter.incrementAndGet();
if (counter.get() > Long.MAX_VALUE - 100) {
counter.set(0);
}
try {
Thread.sleep(10000);
} catch (InterruptedException e) {
log.error(e);
}
synchronized(subscribedClient) {
Iterator<DeferredResult<String>> it = subscribedClient.iterator();
while(it.hasNext()) {
DeferredResult<String> dr = it.next();
dr.setResult("{ \"data\" : \"Deferred Broadcast News # "+ counter +"\" }");
it.remove();
}
}
}
}
});
t.setDaemon(true);
t.setName("BroadcastDeferredThread");
t.start();
}


public void addSubscribed(DeferredResult<String> client) {
synchronized(subscribedClient) {
subscribedClient.add(client);
}
}

}

At the heart of this class is a subscribed client list - where all the deferred results stored, to be processed when new event occurs. Servlets add deferred result to the list when request is received, and the Broadcast Deferred Thread will go through the list and assigns the news to each of the waiting deferred result. The broadcast thread runs indefinitely, incrementing the news counter, then sleeping for 10 seconds and then broadcasting the news to all of the subscribed clients. If, by chance a deferred result is already expired, the call to setResult is ignored by the Spring API.

Here is a sample page with the result from running this task.

Deferred Result Example

Scalability Testing

To compare the scalability of the pre-Servlet 3.0 solution with the asynchronous servlet, I wrote a program that creates multiple threads, retrieves the web page content, and measures the time it took to perform the retrieve operation (WebClient). It was not essential for the analysis to know in specific time results, e.g. whether it took 9,000 milliseconds vs 10,000 for one call, but rather a trend of what happens when the number of threads accessing the site are increased.  Here are the results testing:

Servlet

# threads

time (milis)

st. dev

pre-Servlet 3.0 spec servlet

68

9,424

2.19

pre-Servlet 3.0 spec servlet

128

9,993

2.33

pre-Servlet 3.0 spec servlet

256

10,149

785.31

pre-Servlet 3.0 spec servlet

512

19,610

898.06

pre-Servlet 3.0 spec servlet

1024

38,679

1,743.00

pre-Servlet 3.0 spec servlet

2048

76,925

3,780.28

Servlet 3.0 asynchronous servlet

68

9,076

4.87

Servlet 3.0 asynchronous servlet

128

10,028

2.77

Servlet 3.0 asynchronous servlet

256

10,034

3.96

Servlet 3.0 asynchronous servlet

512

10,014

9.44

Servlet 3.0 asynchronous servlet

1024

10,027

21.57

Servlet 3.0 asynchronous servlet

2048

10,014

39.40







 

Performance Test

As expected, as the number of threads accessing synchronous long-polling solution is increased, the response time increases. Up until just under 256 thread both synchronous and asynchronous solution performance is very similar. It is most likely because the web server (jetty jetty-8.1.13.v20130916) probably has that many threads servicing requests, so as requests are coming from the test program, almost none have to wait to be serviced. After 256 threads, the difference between the two solutions is highly noticeable.

As you look at the test data for synchronous service at or after 256 threads, two notable things happen: the response time increases with number of threads, and the standard deviation also increases. This increase in both time to get response and standard deviation happens because there are less threads available to process client requests as the number of clients increase. Depending on when a particular request comes in, all threads on the server might be already busy waiting for the news update. In this situation, the amount of time the new request has to wait depends on not only when it came in, e.g. 0.1 second or 0.2 seconds before the news are available to be broadcasted, but also how many threads are waiting in line before it could even get to be serviced.

The asynchronous test results shows remarkable difference in comparison. There is no ramp up in the response time as the number of requests grow, and one can observe only relatively small standard deviation increase. The results suggest that as requests are coming, they are all getting serviced by being put aside to wait for the news update, while server threads that service requests are returned back to service other user requests. So, if you need to use long polling and your server supports Servlet 3.0 specification use asynchronous servlets since they are scaling better.

p>Realistic example News Publishing Subscribe site.

 

The last example is a little more interactive, it allows one person to subscribe to news on one of four channels and another person can post news on these channels. There is also an auto-posting service that posts news as well.

Client Example

News Service Example

At the heart of this service is the NewsDistributionEngine class. It has three methods: addMessageConsumer, removeMessageConsumer and postMessage. AddMessageConsumer adds a consumer request (a DeferredResult) to the list of consumers interested in the news topic: this is done when the AJAX is polling the news service after user had subscribed to a news topic. Lists of consumers interested in specific topics is held in a map, which links topic id to the list of interested consumers. removeMessageConsumer removes the consumer from that list. PostMessage is called by clients that posts news: either the web page or an automatic broadcasting client. To process the news and update all clients, application starts a new thread that first gets the list of the clients subscribed to the news and, if the request did not timed out, posts the news to the client. Clients are then removed from subscribed clients, since subscription only lasts for one request. The client will re-subscribe to the topic via long polling AJAX.

SubscribeToNews is a spring MVC component, that has 3 methods: show the subscription page, process subscription and poll for news. It utilizes HttpSession to remember last subscription for the client, which is then used in the poll for news method to subscribe to the appropriate news channel. PublishNewsController is also a spring MVC component which shows the news posing page and when news are posted, calls NewsDistributionEngine to process the news.

Conclusion
The article has shown three examples of long-polling applications: one using pre-Servlet 3.0 spec, second one using Servlet 3.0 asynchronous servelts and third example which is a bit more interactive. In these examples I showed how implement long polling on both server and client side. As you choose the implementation of long polling, the results show that using asynchronous servlets will scale much better synchronous servlets. I hope you can find these examples useful in understanding how to implement long polling. The code used in these examples could be found at ... (where do you usually like to get sources, git?)?

More Stories By Henry Naftulin

Henry Naftulin is seasoned, result-oriented enterprise senior software developer with 10 years of experience in enterprise scale software applications. He is highly skilled in Java (J2EE), Web and database enterprise application development. Recognized as strong team player, technical leader and mentor for junior team members. Latest interests include e-Logistics/Supply Chain, financial applications and performance tuning.

@CloudExpo Stories
"Our strategy is to focus on the hyperscale providers - AWS, Azure, and Google. Over the last year we saw that a lot of developers need to learn how to do their job in the cloud and we see this DevOps movement that we are catering to with our content," stated Alessandro Fasan, Head of Global Sales at Cloud Academy, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
As organizations shift towards IT-as-a-service models, the need for managing and protecting data residing across physical, virtual, and now cloud environments grows with it. Commvault can ensure protection, access and E-Discovery of your data – whether in a private cloud, a Service Provider delivered public cloud, or a hybrid cloud environment – across the heterogeneous enterprise. In his general session at 18th Cloud Expo, Randy De Meno, Chief Technologist - Windows Products and Microsoft Part...
DXWorldEXPO LLC announced today that ICC-USA, a computer systems integrator and server manufacturing company focused on developing products and product appliances, will exhibit at the 22nd International CloudEXPO | DXWorldEXPO. DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City. ICC is a computer systems integrator and server manufacturing company focused on developing products and product appliances to meet a wide range of ...
Andi Mann, Chief Technology Advocate at Splunk, is an accomplished digital business executive with extensive global expertise as a strategist, technologist, innovator, marketer, and communicator. For over 30 years across five continents, he has built success with Fortune 500 corporations, vendors, governments, and as a leading research analyst and consultant.
JETRO showcased Japan Digital Transformation Pavilion at SYS-CON's 21st International Cloud Expo® at the Santa Clara Convention Center in Santa Clara, CA. The Japan External Trade Organization (JETRO) is a non-profit organization that provides business support services to companies expanding to Japan. With the support of JETRO's dedicated staff, clients can incorporate their business; receive visa, immigration, and HR support; find dedicated office space; identify local government subsidies; get...
René Bostic is the Technical VP of the IBM Cloud Unit in North America. Enjoying her career with IBM during the modern millennial technological era, she is an expert in cloud computing, DevOps and emerging cloud technologies such as Blockchain. Her strengths and core competencies include a proven record of accomplishments in consensus building at all levels to assess, plan, and implement enterprise and cloud computing solutions. René is a member of the Society of Women Engineers (SWE) and a m...
In this presentation, you will learn first hand what works and what doesn't while architecting and deploying OpenStack. Some of the topics will include:- best practices for creating repeatable deployments of OpenStack- multi-site considerations- how to customize OpenStack to integrate with your existing systems and security best practices.
Explosive growth in connected devices. Enormous amounts of data for collection and analysis. Critical use of data for split-second decision making and actionable information. All three are factors in making the Internet of Things a reality. Yet, any one factor would have an IT organization pondering its infrastructure strategy. How should your organization enhance its IT framework to enable an Internet of Things implementation? In his session at @ThingsExpo, James Kirkland, Red Hat's Chief Archi...
Digital transformation has increased the pace of business creating a productivity divide between the technology haves and have nots. Managing financial information on spreadsheets and piecing together insight from numerous disconnected systems is no longer an option. Rapid market changes and aggressive competition are motivating business leaders to reevaluate legacy technology investments in search of modern technologies to achieve greater agility, reduced costs and organizational efficiencies. ...
"With Digital Experience Monitoring what used to be a simple visit to a web page has exploded into app on phones, data from social media feeds, competitive benchmarking - these are all components that are only available because of some type of digital asset," explained Leo Vasiliou, Director of Web Performance Engineering at Catchpoint Systems, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
It is ironic, but perhaps not unexpected, that many organizations who want the benefits of using an Agile approach to deliver software use a waterfall approach to adopting Agile practices: they form plans, they set milestones, and they measure progress by how many teams they have engaged. Old habits die hard, but like most waterfall software projects, most waterfall-style Agile adoption efforts fail to produce the results desired. The problem is that to get the results they want, they have to ch...
Organizations planning enterprise data center consolidation and modernization projects are faced with a challenging, costly reality. Requirements to deploy modern, cloud-native applications simultaneously with traditional client/server applications are almost impossible to achieve with hardware-centric enterprise infrastructure. Compute and network infrastructure are fast moving down a software-defined path, but storage has been a laggard. Until now.
Without a clear strategy for cost control and an architecture designed with cloud services in mind, costs and operational performance can quickly get out of control. To avoid multiple architectural redesigns requires extensive thought and planning. Boundary (now part of BMC) launched a new public-facing multi-tenant high resolution monitoring service on Amazon AWS two years ago, facing challenges and learning best practices in the early days of the new service.
HyperConvergence came to market with the objective of being simple, flexible and to help drive down operating expenses. It reduced the footprint by bundling the compute/storage/network into one box. This brought a new set of challenges as the HyperConverged vendors are very focused on their own proprietary building blocks. If you want to scale in a certain way, let's say you identified a need for more storage and want to add a device that is not sold by the HyperConverged vendor, forget about it...
Digital Transformation is much more than a buzzword. The radical shift to digital mechanisms for almost every process is evident across all industries and verticals. This is often especially true in financial services, where the legacy environment is many times unable to keep up with the rapidly shifting demands of the consumer. The constant pressure to provide complete, omnichannel delivery of customer-facing solutions to meet both regulatory and customer demands is putting enormous pressure on...
The best way to leverage your CloudEXPO | DXWorldEXPO presence as a sponsor and exhibitor is to plan your news announcements around our events. The press covering CloudEXPO | DXWorldEXPO will have access to these releases and will amplify your news announcements. More than two dozen Cloud companies either set deals at our shows or have announced their mergers and acquisitions at CloudEXPO. Product announcements during our show provide your company with the most reach through our targeted audienc...
@DevOpsSummit at Cloud Expo, taking place November 12-13 in New York City, NY, is co-located with 22nd international CloudEXPO | first international DXWorldEXPO and will feature technical sessions from a rock star conference faculty and the leading industry players in the world.
With 10 simultaneous tracks, keynotes, general sessions and targeted breakout classes, @CloudEXPO and DXWorldEXPO are two of the most important technology events of the year. Since its launch over eight years ago, @CloudEXPO and DXWorldEXPO have presented a rock star faculty as well as showcased hundreds of sponsors and exhibitors!
DXWorldEXPO LLC announced today that the upcoming DXWorldEXPO | CloudEXPO New York event will feature 10 companies from Poland to participate at the "Poland Digital Transformation Pavilion" on November 12-13, 2018.
Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more business becomes digital the more stakeholders are interested in this data including how it relates to business. Some of these people have never used a monitoring tool before. They have a question on their mind like “How is my application doing” but no id...