|By Michael Kopp||
|October 27, 2012 10:00 AM EDT||
Traditional Enterprise Database vendors often bring up the lack of professional monitoring and management tool support for NoSQL solutions. Their argument is that enterprise applications require sophisticated tuning and monitoring of the database in order to ensure a performant and smooth operation. NoSQL Vendors, while arguing that this lack is not enough to favor RDBMS over their respective solutions, do agree. Several vendors try to differentiate themselves by providing enterprise level monitoring and management software, for example, Cassandra, MongoDB, HBase or others. Both are of course correct that monitoring and management of especially the performance aspect is important, but at the same time they are making the same mistake that RDBMS vendors have made for the last decade: they ignore the application.
Application Performance Management for Databases
What matters in the end is not the database performance itself, but the performance of the application that uses the database. We have explained the different problem patterns numerous times in this blog (...) so I won't go into them. All of them however have one thing in common: the application logic drives how the database is used and there is only so much that you can tune on the database to cover for mistakes on the application side. So we need to monitor and optimize that usage pattern itself. Application Logic again is driven by input data or in most cases end user interaction, thus we need to understand how end user behavior and end user actions drive the database usage. On the other hand we need to understand the impact of the database on these actions. What's important to understand is that the database can work and perform to the highest standards and still be the main bottleneck as far as the application is concerned - if it is wrongly used or has a bad access pattern. RDBMS and NoSQL Databases have that in common. Therefore the fundamental way I, as a performance engineer, do application performance analysis and management does not change:
First we need to understand if a particular business transaction slows down, has a general performance problem and if this has an impact on the end user:
Do any of the business transactions violate a baseline or have negative end user experience
Next we would isolate the high level cause of the slow down or performance issue. There are many ways of doing this, but it's always some kind of fault domain isolation:
Transaction Flow shows that we spend ~15 % waiting for the Database
This Transaction Flow shows that the Business Backend is calling a Cassandra Database Cluster
This tells us if we spend time waiting on the database. We see that there is not much difference between a regular database and something like an Apache Cassandra!
What's important here is that if the database shows up as the main contributor, this does not mean that the database itself is at fault, it might just be the applications usage of it. Thus I would need to check the usage and access pattern next:
This shows the select statements executed within a particular transaction type
This shows all Cassandra database statements against all participating Cassandra servers executed in a particular transaction
Here I might see that the reason for bad performance is that we execute too many statements per transaction, or that we read too much data. If that is the case I would need to check the application logic itself and potentially need a developer to fix it. The developer would of course want to understand where, why and in which particular transaction the statements are executed.
This shows a single Transaction (PurePath) and the Cassandra Statements executed within it
If however a particular statement is slow we, might very well have a database issue and I would talk to the DBA. The only difference in case of a NoSQL solution in this process is that you often have a database cluster, so I would want to understand if the problem is isolated to a particular node or not. And the DBA will want to understand if my access pattern leads to a good distribution across that cluster or if I am hammering away at a subset of them.
This hotspot view shows that Cassandra Server Node3 consumes much more Wait and I/O time than the others
The nice thing is that all in all my analysis does not differ between JDBC, ADO, Cassandra or one of the many NoSQL solutions.
APM Solution Support
There is of course a caveat here; it requires some level of support of the APM solution of choice. Sometimes it might be enough to see API calls on the NoSQL client in my response time breakdown. More often than not of course a little bit more context is desired, like which ColumnFamily is accessed and how many rows are read or which Database Node in the Cluster is serving a read. For this and the afore mentioned reasons I argue that APM solution support of your chosen Database or NoSQL solution is as important as the monitoring of the database itself.
I have spent considerable time tuning SQL statements and indexes, but in the end the best optimizations have always been those on the application and how the application uses the database. SQL Tuning almost always adds complexity and often is a workaround over bad application or data structure design. In the NoSQL world "SQL statement" tuning for the most part is a task of the past, but Data Structure Design has retained its importance! At the same time logic that traditionally resided in the database is now in the application layer, making application design even more important than before. So while some things have shifted, from an Application Performance Engineering Perspective I have to say: nothing really changed, it's still about the application. Now more than ever!
With an estimated 50 billion devices connected to the Internet by 2020, several industries will begin to expand their capabilities for retaining end point data at the edge to better utilize the range of data types and sheer volume of M2M data generated by the Internet of Things. In his session at @ThingsExpo, Don DeLoach, CEO and President of Infobright, will discuss the infrastructures businesses will need to implement to handle this explosion of data by providing specific use cases for filte...
Feb. 8, 2016 08:00 PM EST Reads: 131
SYS-CON Events announced today that Pythian, a global IT services company specializing in helping companies adopt disruptive technologies to optimize revenue-generating systems, has been named “Bronze Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2015 at the Javits Center in New York, New York. Founded in 1997, Pythian is a global IT services company that helps companies compete by adopting disruptive technologies such as cloud, Big Data, advanced analytics, and DevO...
Feb. 8, 2016 06:30 PM EST Reads: 167
SYS-CON Events announced today that VAI, a leading ERP software provider, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. VAI (Vormittag Associates, Inc.) is a leading independent mid-market ERP software developer renowned for its flexible solutions and ability to automate critical business functions for the distribution, manufacturing, specialty retail and service sectors. An IBM Premier Business Part...
Feb. 8, 2016 03:00 PM EST Reads: 574
SYS-CON Events announced today that Catchpoint Systems, Inc., a provider of innovative web and infrastructure monitoring solutions, has been named “Silver Sponsor” of SYS-CON's DevOps Summit at 18th Cloud Expo New York, which will take place June 7-9, 2016, at the Javits Center in New York City, NY. Catchpoint is a leading Digital Performance Analytics company that provides unparalleled insight into customer-critical services to help consistently deliver an amazing customer experience. Designed...
Feb. 8, 2016 02:00 PM EST Reads: 342
SYS-CON Events announced today that Alert Logic, Inc., the leading provider of Security-as-a-Service solutions for the cloud, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Alert Logic, Inc., provides Security-as-a-Service for on-premises, cloud, and hybrid infrastructures, delivering deep security insight and continuous protection for customers at a lower cost than traditional security solutions. Ful...
Feb. 8, 2016 02:00 PM EST Reads: 378
In most cases, it is convenient to have some human interaction with a web (micro-)service, no matter how small it is. A traditional approach would be to create an HTTP interface, where user requests will be dispatched and HTML/CSS pages must be served. This approach is indeed very traditional for a web site, but not really convenient for a web service, which is not intended to be good looking, 24x7 up and running and UX-optimized. Instead, talking to a web service in a chat-bot mode would be muc...
Feb. 8, 2016 02:00 PM EST Reads: 210
[session] From Build to Scale: Lifecycle of Microservices By @fortyfivan | @CloudExpo #Microservices
More and more companies are looking to microservices as an architectural pattern for breaking apart applications into more manageable pieces so that agile teams can deliver new features quicker and more effectively. What this pattern has done more than anything to date is spark organizational transformations, setting the foundation for future application development. In practice, however, there are a number of considerations to make that go beyond simply “build, ship, and run,” which changes ho...
Feb. 8, 2016 01:30 PM EST Reads: 175
SYS-CON Events announced today that Interoute, owner-operator of one of Europe's largest networks and a global cloud services platform, has been named “Bronze Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2015 at the Javits Center in New York, New York. Interoute is the owner-operator of one of Europe's largest networks and a global cloud services platform which encompasses 12 data centers, 14 virtual data centers and 31 colocation centers, with connections to 195 ad...
Feb. 8, 2016 12:45 PM EST Reads: 357
Advances in technology and ubiquitous connectivity have made the utilization of a dispersed workforce more common. Whether that remote team is located across the street or country, management styles/ approaches will have to be adjusted to accommodate this new dynamic. In his session at 17th Cloud Expo, Sagi Brody, Chief Technology Officer at Webair Internet Development Inc., focused on the challenges of managing remote teams, providing real-world examples that demonstrate what works and what do...
Feb. 8, 2016 12:30 PM EST Reads: 237
[session] Focusing on Time-to-Value in Big Data Deployments By @AndyWarfield | @BigDataExpo #BigData
As enterprises work to take advantage of Big Data technologies, they frequently become distracted by product-level decisions. In most new Big Data builds this approach is completely counter-productive: it presupposes tools that may not be a fit for development teams, forces IT to take on the burden of evaluating and maintaining unfamiliar technology, and represents a major up-front expense. In his session at @BigDataExpo at @ThingsExpo, Andrew Warfield, CTO and Co-Founder of Coho Data, will dis...
Feb. 8, 2016 12:30 PM EST Reads: 145
SYS-CON Events announced today that Men & Mice, the leading global provider of DNS, DHCP and IP address management overlay solutions, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. The Men & Mice Suite overlay solution is already known for its powerful application in heterogeneous operating environments, enabling enterprises to scale without fuss. Building on a solid range of diverse platform support,...
Feb. 8, 2016 12:00 PM EST Reads: 160
SYS-CON Events announced today that Commvault, a global leader in enterprise data protection and information management, has been named “Bronze Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY, and the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Commvault is a leading provider of data protection and information management...
Feb. 8, 2016 10:45 AM EST Reads: 384
SYS-CON Events announced today that AppNeta, the leader in performance insight for business-critical web applications, will exhibit and present at SYS-CON's @DevOpsSummit at Cloud Expo New York, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. AppNeta is the only application performance monitoring (APM) company to provide solutions for all applications – applications you develop internally, business-critical SaaS applications you use and the networks that deli...
Feb. 8, 2016 10:15 AM EST Reads: 366
It's easy to assume that your app will run on a fast and reliable network. The reality for your app's users, though, is often a slow, unreliable network with spotty coverage. What happens when the network doesn't work, or when the device is in airplane mode? You get unhappy, frustrated users. An offline-first app is an app that works, without error, when there is no network connection.
Feb. 8, 2016 09:45 AM EST Reads: 172
The cloud promises new levels of agility and cost-savings for Big Data, data warehousing and analytics. But it’s challenging to understand all the options – from IaaS and PaaS to newer services like HaaS (Hadoop as a Service) and BDaaS (Big Data as a Service). In her session at @BigDataExpo at @ThingsExpo, Hannah Smalltree, a director at Cazena, will provide an educational overview of emerging “as-a-service” options for Big Data in the cloud. This is critical background for IT and data profes...
Feb. 8, 2016 09:30 AM EST Reads: 159
Father business cycles and digital consumers are forcing enterprises to respond faster to customer needs and competitive demands. Successful integration of DevOps and Agile development will be key for business success in today’s digital economy. In his session at DevOps Summit, Pradeep Prabhu, Co-Founder & CEO of Cloudmunch, covered the critical practices that enterprises should consider to seamlessly integrate Agile and DevOps processes, barriers to implementing this in the enterprise, and pr...
Feb. 8, 2016 09:30 AM EST Reads: 339
Cognitive Computing is becoming the foundation for a new generation of solutions that have the potential to transform business. Unlike traditional approaches to building solutions, a cognitive computing approach allows the data to help determine the way applications are designed. This contrasts with conventional software development that begins with defining logic based on the current way a business operates. In her session at 18th Cloud Expo, Judith S. Hurwitz, President and CEO of Hurwitz & ...
Feb. 8, 2016 09:15 AM EST Reads: 204
How Best to Integrate Cloud Foundry into Your Existing Ecosystem By @Gidrontxt | @DevOpsSummit #DevOps
As someone who has been dedicated to automation and Application Release Automation (ARA) technology for almost six years now, one of the most common questions I get asked regards Platform-as-a-Service (PaaS). Specifically, people want to know whether release automation is still needed when a PaaS is in place, and why. Isn't that what a PaaS provides? A solution to the deployment and runtime challenges of an application? Why would anyone using a PaaS then need an automation engine with workflow ...
Feb. 8, 2016 07:00 AM EST Reads: 149
SYS-CON Events announced today that Fusion, a leading provider of cloud services, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Fusion, a leading provider of integrated cloud solutions to small, medium and large businesses, is the industry's single source for the cloud. Fusion's advanced, proprietary cloud service platform enables the integration of leading edge solutions in the cloud, including clou...
Feb. 6, 2016 03:30 PM EST Reads: 739
Your business relies on your applications and your employees to stay in business. Whether you develop apps or manage business critical apps that help fuel your business, what happens when users experience sluggish performance? You and all technical teams across the organization – application, network, operations, among others, as well as, those outside the organization, like ISPs and third-party providers – are called in to solve the problem.
Feb. 6, 2016 02:00 PM EST Reads: 699