|By Michael Kopp||
|October 27, 2012 10:00 AM EDT||
Traditional Enterprise Database vendors often bring up the lack of professional monitoring and management tool support for NoSQL solutions. Their argument is that enterprise applications require sophisticated tuning and monitoring of the database in order to ensure a performant and smooth operation. NoSQL Vendors, while arguing that this lack is not enough to favor RDBMS over their respective solutions, do agree. Several vendors try to differentiate themselves by providing enterprise level monitoring and management software, for example, Cassandra, MongoDB, HBase or others. Both are of course correct that monitoring and management of especially the performance aspect is important, but at the same time they are making the same mistake that RDBMS vendors have made for the last decade: they ignore the application.
Application Performance Management for Databases
What matters in the end is not the database performance itself, but the performance of the application that uses the database. We have explained the different problem patterns numerous times in this blog (...) so I won't go into them. All of them however have one thing in common: the application logic drives how the database is used and there is only so much that you can tune on the database to cover for mistakes on the application side. So we need to monitor and optimize that usage pattern itself. Application Logic again is driven by input data or in most cases end user interaction, thus we need to understand how end user behavior and end user actions drive the database usage. On the other hand we need to understand the impact of the database on these actions. What's important to understand is that the database can work and perform to the highest standards and still be the main bottleneck as far as the application is concerned - if it is wrongly used or has a bad access pattern. RDBMS and NoSQL Databases have that in common. Therefore the fundamental way I, as a performance engineer, do application performance analysis and management does not change:
First we need to understand if a particular business transaction slows down, has a general performance problem and if this has an impact on the end user:
Do any of the business transactions violate a baseline or have negative end user experience
Next we would isolate the high level cause of the slow down or performance issue. There are many ways of doing this, but it's always some kind of fault domain isolation:
Transaction Flow shows that we spend ~15 % waiting for the Database
This Transaction Flow shows that the Business Backend is calling a Cassandra Database Cluster
This tells us if we spend time waiting on the database. We see that there is not much difference between a regular database and something like an Apache Cassandra!
What's important here is that if the database shows up as the main contributor, this does not mean that the database itself is at fault, it might just be the applications usage of it. Thus I would need to check the usage and access pattern next:
This shows the select statements executed within a particular transaction type
This shows all Cassandra database statements against all participating Cassandra servers executed in a particular transaction
Here I might see that the reason for bad performance is that we execute too many statements per transaction, or that we read too much data. If that is the case I would need to check the application logic itself and potentially need a developer to fix it. The developer would of course want to understand where, why and in which particular transaction the statements are executed.
This shows a single Transaction (PurePath) and the Cassandra Statements executed within it
If however a particular statement is slow we, might very well have a database issue and I would talk to the DBA. The only difference in case of a NoSQL solution in this process is that you often have a database cluster, so I would want to understand if the problem is isolated to a particular node or not. And the DBA will want to understand if my access pattern leads to a good distribution across that cluster or if I am hammering away at a subset of them.
This hotspot view shows that Cassandra Server Node3 consumes much more Wait and I/O time than the others
The nice thing is that all in all my analysis does not differ between JDBC, ADO, Cassandra or one of the many NoSQL solutions.
APM Solution Support
There is of course a caveat here; it requires some level of support of the APM solution of choice. Sometimes it might be enough to see API calls on the NoSQL client in my response time breakdown. More often than not of course a little bit more context is desired, like which ColumnFamily is accessed and how many rows are read or which Database Node in the Cluster is serving a read. For this and the afore mentioned reasons I argue that APM solution support of your chosen Database or NoSQL solution is as important as the monitoring of the database itself.
I have spent considerable time tuning SQL statements and indexes, but in the end the best optimizations have always been those on the application and how the application uses the database. SQL Tuning almost always adds complexity and often is a workaround over bad application or data structure design. In the NoSQL world "SQL statement" tuning for the most part is a task of the past, but Data Structure Design has retained its importance! At the same time logic that traditionally resided in the database is now in the application layer, making application design even more important than before. So while some things have shifted, from an Application Performance Engineering Perspective I have to say: nothing really changed, it's still about the application. Now more than ever!
"We build IoT infrastructure products - when you have to integrate different devices, different systems and cloud you have to build an application to do that but we eliminate the need to build an application. Our products can integrate any device, any system, any cloud regardless of protocol," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Dec. 4, 2016 02:15 AM EST Reads: 840
When it comes to cloud computing, the ability to turn massive amounts of compute cores on and off on demand sounds attractive to IT staff, who need to manage peaks and valleys in user activity. With cloud bursting, the majority of the data can stay on premises while tapping into compute from public cloud providers, reducing risk and minimizing need to move large files. In his session at 18th Cloud Expo, Scott Jeschonek, Director of Product Management at Avere Systems, discussed the IT and busin...
Dec. 4, 2016 02:00 AM EST Reads: 3,779
Between 2005 and 2020, data volumes will grow by a factor of 300 – enough data to stack CDs from the earth to the moon 162 times. This has come to be known as the ‘big data’ phenomenon. Unfortunately, traditional approaches to handling, storing and analyzing data aren’t adequate at this scale: they’re too costly, slow and physically cumbersome to keep up. Fortunately, in response a new breed of technology has emerged that is cheaper, faster and more scalable. Yet, in meeting these new needs they...
Dec. 4, 2016 12:30 AM EST Reads: 1,780
The cloud promises new levels of agility and cost-savings for Big Data, data warehousing and analytics. But it’s challenging to understand all the options – from IaaS and PaaS to newer services like HaaS (Hadoop as a Service) and BDaaS (Big Data as a Service). In her session at @BigDataExpo at @ThingsExpo, Hannah Smalltree, a director at Cazena, provided an educational overview of emerging “as-a-service” options for Big Data in the cloud. This is critical background for IT and data professionals...
Dec. 3, 2016 11:00 PM EST Reads: 4,142
"Once customers get a year into their IoT deployments, they start to realize that they may have been shortsighted in the ways they built out their deployment and the key thing I see a lot of people looking at is - how can I take equipment data, pull it back in an IoT solution and show it in a dashboard," stated Dave McCarthy, Director of Products at Bsquare Corporation, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Dec. 3, 2016 11:00 PM EST Reads: 952
@DevOpsSummit taking place June 6-8, 2017 at Javits Center, New York City, is co-located with the 20th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. @DevOpsSummit at Cloud Expo New York Call for Papers is now open.
Dec. 3, 2016 09:30 PM EST Reads: 1,767
The cloud competition for database hosts is fierce. How do you evaluate a cloud provider for your database platform? In his session at 18th Cloud Expo, Chris Presley, a Solutions Architect at Pythian, gave users a checklist of considerations when choosing a provider. Chris Presley is a Solutions Architect at Pythian. He loves order – making him a premier Microsoft SQL Server expert. Not only has he programmed and administered SQL Server, but he has also shared his expertise and passion with b...
Dec. 3, 2016 08:00 PM EST Reads: 3,956
In his session at 19th Cloud Expo, Claude Remillard, Principal Program Manager in Developer Division at Microsoft, contrasted how his team used config as code and immutable patterns for continuous delivery of microservices and apps to the cloud. He showed how the immutable patterns helps developers do away with most of the complexity of config as code-enabling scenarios such as rollback, zero downtime upgrades with far greater simplicity. He also demoed building immutable pipelines in the cloud ...
Dec. 3, 2016 08:00 PM EST Reads: 1,746
As data explodes in quantity, importance and from new sources, the need for managing and protecting data residing across physical, virtual, and cloud environments grow with it. Managing data includes protecting it, indexing and classifying it for true, long-term management, compliance and E-Discovery. Commvault can ensure this with a single pane of glass solution – whether in a private cloud, a Service Provider delivered public cloud or a hybrid cloud environment – across the heterogeneous enter...
Dec. 3, 2016 06:15 PM EST Reads: 1,515
"IoT is going to be a huge industry with a lot of value for end users, for industries, for consumers, for manufacturers. How can we use cloud to effectively manage IoT applications," stated Ian Khan, Innovation & Marketing Manager at Solgeniakhela, in this SYS-CON.tv interview at @ThingsExpo, held November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA.
Dec. 3, 2016 05:30 PM EST Reads: 4,047
Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more business becomes digital the more stakeholders are interested in this data including how it relates to business. Some of these people have never used a monitoring tool before. They have a question on their mind like “How is my application doing” but no id...
Dec. 3, 2016 05:15 PM EST Reads: 2,139
@GonzalezCarmen has been ranked the Number One Influencer and @ThingsExpo has been named the Number One Brand in the “M2M 2016: Top 100 Influencers and Brands” by Onalytica. Onalytica analyzed tweets over the last 6 months mentioning the keywords M2M OR “Machine to Machine.” They then identified the top 100 most influential brands and individuals leading the discussion on Twitter.
Dec. 3, 2016 05:15 PM EST Reads: 2,004
In IT, we sometimes coin terms for things before we know exactly what they are and how they’ll be used. The resulting terms may capture a common set of aspirations and goals – as “cloud” did broadly for on-demand, self-service, and flexible computing. But such a term can also lump together diverse and even competing practices, technologies, and priorities to the point where important distinctions are glossed over and lost.
Dec. 3, 2016 04:30 PM EST Reads: 1,477
Predictive analytics tools monitor, report, and troubleshoot in order to make proactive decisions about the health, performance, and utilization of storage. Most enterprises combine cloud and on-premise storage, resulting in blended environments of physical, virtual, cloud, and other platforms, which justifies more sophisticated storage analytics. In his session at 18th Cloud Expo, Peter McCallum, Vice President of Datacenter Solutions at FalconStor, discussed using predictive analytics to mon...
Dec. 3, 2016 04:00 PM EST Reads: 4,864
All clouds are not equal. To succeed in a DevOps context, organizations should plan to develop/deploy apps across a choice of on-premise and public clouds simultaneously depending on the business needs. This is where the concept of the Lean Cloud comes in - resting on the idea that you often need to relocate your app modules over their life cycles for both innovation and operational efficiency in the cloud. In his session at @DevOpsSummit at19th Cloud Expo, Valentin (Val) Bercovici, CTO of Soli...
Dec. 3, 2016 03:30 PM EST Reads: 1,595
Information technology is an industry that has always experienced change, and the dramatic change sweeping across the industry today could not be truthfully described as the first time we've seen such widespread change impacting customer investments. However, the rate of the change, and the potential outcomes from today's digital transformation has the distinct potential to separate the industry into two camps: Organizations that see the change coming, embrace it, and successful leverage it; and...
Dec. 3, 2016 03:15 PM EST Reads: 3,227
Without a clear strategy for cost control and an architecture designed with cloud services in mind, costs and operational performance can quickly get out of control. To avoid multiple architectural redesigns requires extensive thought and planning. Boundary (now part of BMC) launched a new public-facing multi-tenant high resolution monitoring service on Amazon AWS two years ago, facing challenges and learning best practices in the early days of the new service. In his session at 19th Cloud Exp...
Dec. 3, 2016 03:00 PM EST Reads: 500
Extracting business value from Internet of Things (IoT) data doesn’t happen overnight. There are several requirements that must be satisfied, including IoT device enablement, data analysis, real-time detection of complex events and automated orchestration of actions. Unfortunately, too many companies fall short in achieving their business goals by implementing incomplete solutions or not focusing on tangible use cases. In his general session at @ThingsExpo, Dave McCarthy, Director of Products...
Dec. 3, 2016 02:45 PM EST Reads: 533
Regulatory requirements exist to promote the controlled sharing of information, while protecting the privacy and/or security of the information. Regulations for each type of information have their own set of rules, policies, and guidelines. Cloud Service Providers (CSP) are faced with increasing demand for services at decreasing prices. Demonstrating and maintaining compliance with regulations is a nontrivial task and doing so against numerous sets of regulatory requirements can be daunting task...
Dec. 3, 2016 02:45 PM EST Reads: 729
Fact: storage performance problems have only gotten more complicated, as applications not only have become largely virtualized, but also have moved to cloud-based infrastructures. Storage performance in virtualized environments isn’t just about IOPS anymore. Instead, you need to guarantee performance for individual VMs, helping applications maintain performance as the number of VMs continues to go up in real time. In his session at Cloud Expo, Dhiraj Sehgal, Product and Marketing at Tintri, sha...
Dec. 3, 2016 02:45 PM EST Reads: 790