Welcome!

Machine Learning Authors: Yeshim Deniz, Pat Romanski, Liz McMillan, Elizabeth White, Corey Roth

Related Topics: Java IoT, Industrial IoT, Open Source Cloud, Machine Learning , Apache, SDN Journal

Java IoT: Article

Tracing Black Boxes III: Solr Query Performance Tuning

Solr Query Performance Tuning

Solr Server provides JMX statistics that show performance details such as query speed and cache hit/miss rates in a macro level, which James talked about in a previous post. However, it might be tough to trace how a particular operation; for example: how a specific query fared in the system. This week, I'd like to introduce TraceView's latest support for Solr Server, which provides breakdown of each operation, enabling more precise performance monitoring and troubleshooting.

How does Solr work, anyway?

Requests made to Solr Server can be roughly divided into 3 categories: queries, data indexing/update, admin operations (check server health/log, optimize server etc). Requests made to Solr server first go through the SolrDispatchFilter that looks up the corresponding Solr Core (a running instance of index/dataset along with configurations) and handler. The retrieved SolrCore and handler are then used to process the request. For SearchHandler, it is further broken down into smaller SearchComponents (for example, querying, highlighting results, calculating statistics etc).

Take note that since version 4, Solr added distributed capabilities in Solr called SolrCloud, which enable highly available, fault tolerant cluster of Solr servers. Index/dataset is no longer hosted on a single Core/instance. Instead, it is constructed by several shards, each as a disjoint subset of the documents in the index, and each shard can be backed by multiple cores, which are replicas of each other.

For caching, Solr uses several built-in classes (LRUCache, FastLRUCache, LFUCache) for various aspects (field, filter, query result, document etc). They are implemented as in-memory caches to allow quicker operations. The configurations of caches can be found in the Query section of solrconfig.xml.

Indexing and updates can be slow, and certainly compete for resources, but they can be done asynchronously. Admin operations can be slow as well, but they are typically one-time events, or not performance sensitive. Queries are typically the most important operation, as they live in the critical path for the user. TraceView monitor activities on each of the classes that correspond to these actions and their components to extract run-time information on each of the modules described. Let's take a look at an example.

Tracing Queries
A trace follows the path of a request through the entire application. For example, a trace below shows the breakdowns of a query operation:

solr3-request

There's a lot more going on that just searching the index for the given phrase!

Solr has a lot more functionality than just returning a list of documents that match a query. In particular, this query's time is actually dominated by "highlight.process" (highlight matching phrases in the query result) and "ResponseWriter.write" (write data to the response stream). We could speed up this request by nearly 2x by just disabling the highlighter in config!

The other interesting information here is what's not visible. This query didn't hit a cache of any sort. Even for highly variable search terms, the cache can help with users paging through longer lists or sharing links. In this case, TraceView found several cache miss events on "documentCache" that contribute to longer load time.

solr3-cache-info

Pre-warming the cache could help with this, or if this is a common term, it's possible that it's being evicted due to small cache size. We could verify this by checking the JMX statistics for cache eviction rates and adjusting the cache size in our config, if necessary.

Beyond Solr
Instrumentation on Solr Server does not only give an isolated view of how Solr performs, it also draws a complete picture of all the interactions with other applications/systems. For example, below is trace of a query request issued a typical Drupal search page backed by Solr service runs on a different host:

solr3-request-full

We can trace the request handling all the way through the Apache server, Drupal modules, and Solr handling all in one trace. In particular, this tracks requests even when the app and Solr are running on different machines, which allows you to track down issues where the application calls the Solr server incorrectly.

More Stories By Patson Luk

A Java developer who has spent the better part of the last decade working on financial services applications with companies from HSBC to Mobilearth and Parasoft, Patson is experienced in various aspects of computer systems, from large scale enterprise banking system to lightweight mobile payment solutions. He now leads Java instrumentation and tool development for the TraceView product at AppNeta. Patson's focus is on using java bytecode manipulation technologies to gain greater visibility into the full spectrum of Java based technologies. This includes higher level application frameworks from Spring and Struts to Webflow, AppServers from TomCat to JBoss, and Databases from MySQL to Oracle. He's also writes frequently on the AppNeta blog - www.appneta.com/blog

@CloudExpo Stories
In an era of historic innovation fueled by unprecedented access to data and technology, the low cost and risk of entering new markets has leveled the playing field for business. Today, any ambitious innovator can easily introduce a new application or product that can reinvent business models and transform the client experience. In their Day 2 Keynote at 19th Cloud Expo, Mercer Rowe, IBM Vice President of Strategic Alliances, and Raejeanne Skillern, Intel Vice President of Data Center Group and ...
More and more brands have jumped on the IoT bandwagon. We have an excess of wearables – activity trackers, smartwatches, smart glasses and sneakers, and more that track seemingly endless datapoints. However, most consumers have no idea what “IoT” means. Creating more wearables that track data shouldn't be the aim of brands; delivering meaningful, tangible relevance to their users should be. We're in a period in which the IoT pendulum is still swinging. Initially, it swung toward "smart for smart...
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, led attendees through the exciting evolution of the cloud. He looked at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering m...
DXWorldEXPO LLC announced today that All in Mobile, a mobile app development company from Poland, will exhibit at the 22nd International CloudEXPO | DXWorldEXPO. All In Mobile is a mobile app development company from Poland. Since 2014, they maintain passion for developing mobile applications for enterprises and startups worldwide.
@DevOpsSummit at Cloud Expo, taking place November 12-13 in New York City, NY, is co-located with 22nd international CloudEXPO | first international DXWorldEXPO and will feature technical sessions from a rock star conference faculty and the leading industry players in the world.
In his keynote at 19th Cloud Expo, Sheng Liang, co-founder and CEO of Rancher Labs, discussed the technological advances and new business opportunities created by the rapid adoption of containers. With the success of Amazon Web Services (AWS) and various open source technologies used to build private clouds, cloud computing has become an essential component of IT strategy. However, users continue to face challenges in implementing clouds, as older technologies evolve and newer ones like Docker c...
We all know that end users experience the internet primarily with mobile devices. From an app development perspective, we know that successfully responding to the needs of mobile customers depends on rapid DevOps – failing fast, in short, until the right solution evolves in your customers' relationship to your business. Whether you’re decomposing an SOA monolith, or developing a new application cloud natively, it’s not a question of using microservices - not doing so will be a path to eventual ...
The next XaaS is CICDaaS. Why? Because CICD saves developers a huge amount of time. CD is an especially great option for projects that require multiple and frequent contributions to be integrated. But… securing CICD best practices is an emerging, essential, yet little understood practice for DevOps teams and their Cloud Service Providers. The only way to get CICD to work in a highly secure environment takes collaboration, patience and persistence. Building CICD in the cloud requires rigorous ar...
DXWorldEXPO LLC announced today that ICC-USA, a computer systems integrator and server manufacturing company focused on developing products and product appliances, will exhibit at the 22nd International CloudEXPO | DXWorldEXPO. DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City. ICC is a computer systems integrator and server manufacturing company focused on developing products and product appliances to meet a wide range of ...
Coca-Cola’s Google powered digital signage system lays the groundwork for a more valuable connection between Coke and its customers. Digital signs pair software with high-resolution displays so that a message can be changed instantly based on what the operator wants to communicate or sell. In their Day 3 Keynote at 21st Cloud Expo, Greg Chambers, Global Group Director, Digital Innovation, Coca-Cola, and Vidya Nagarajan, a Senior Product Manager at Google, discussed how from store operations and ...
Sanjeev Sharma Joins November 11-13, 2018 @DevOpsSummit at @CloudEXPO New York Faculty. Sanjeev Sharma is an internationally known DevOps and Cloud Transformation thought leader, technology executive, and author. Sanjeev's industry experience includes tenures as CTO, Technical Sales leader, and Cloud Architect leader. As an IBM Distinguished Engineer, Sanjeev is recognized at the highest levels of IBM's core of technical leaders.
We are seeing a major migration of enterprises applications to the cloud. As cloud and business use of real time applications accelerate, legacy networks are no longer able to architecturally support cloud adoption and deliver the performance and security required by highly distributed enterprises. These outdated solutions have become more costly and complicated to implement, install, manage, and maintain.SD-WAN offers unlimited capabilities for accessing the benefits of the cloud and Internet. ...
As Cybric's Chief Technology Officer, Mike D. Kail is responsible for the strategic vision and technical direction of the platform. Prior to founding Cybric, Mike was Yahoo's CIO and SVP of Infrastructure, where he led the IT and Data Center functions for the company. He has more than 24 years of IT Operations experience with a focus on highly-scalable architectures.
Headquartered in Plainsboro, NJ, Synametrics Technologies has provided IT professionals and computer systems developers since 1997. Based on the success of their initial product offerings (WinSQL and DeltaCopy), the company continues to create and hone innovative products that help its customers get more from their computer applications, databases and infrastructure. To date, over one million users around the world have chosen Synametrics solutions to help power their accelerated business or per...
Founded in 2000, Chetu Inc. is a global provider of customized software development solutions and IT staff augmentation services for software technology providers. By providing clients with unparalleled niche technology expertise and industry experience, Chetu has become the premiere long-term, back-end software development partner for start-ups, SMBs, and Fortune 500 companies. Chetu is headquartered in Plantation, Florida, with thirteen offices throughout the U.S. and abroad.
Dion Hinchcliffe is an internationally recognized digital expert, bestselling book author, frequent keynote speaker, analyst, futurist, and transformation expert based in Washington, DC. He is currently Chief Strategy Officer at the industry-leading digital strategy and online community solutions firm, 7Summits.
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science," is responsible for setting the strategy and defining the Big Data service offerings and capabilities for EMC Global Services Big Data Practice. As the CTO for the Big Data Practice, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He's written several white papers, is an avid blogge...
DXWorldEXPO LLC announced today that Dez Blanchfield joined the faculty of CloudEXPO's "10-Year Anniversary Event" which will take place on November 11-13, 2018 in New York City. Dez is a strategic leader in business and digital transformation with 25 years of experience in the IT and telecommunications industries developing strategies and implementing business initiatives. He has a breadth of expertise spanning technologies such as cloud computing, big data and analytics, cognitive computing, m...
"Venafi has a platform that allows you to manage, centralize and automate the complete life cycle of keys and certificates within the organization," explained Gina Osmond, Sr. Field Marketing Manager at Venafi, in this SYS-CON.tv interview at DevOps at 19th Cloud Expo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
"DivvyCloud as a company set out to help customers automate solutions to the most common cloud problems," noted Jeremy Snyder, VP of Business Development at DivvyCloud, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science," is responsible for setting the strategy and defining the Big Data service offerings and capabilities for EMC Global Services Big Data Practice. As the CTO for the Big Data Practice, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He's written several white papers, is an avid blogge...