Machine Learning Authors: Kevin Jackson, Madhavan Krishnan, VP, Cloud Solutions, Virtusa, Pat Romanski, Elizabeth White, William Schmarzo

Related Topics: Microservices Expo, Java IoT, Microsoft Cloud, Containers Expo Blog, Machine Learning , Agile Computing, @CloudExpo, @DXWorldExpo

Microservices Expo: Article

Escape Application Performance "Groundhog Day" with Culture Change

When the problem of performance is addressed across the lifecycle true change can be made

I love movies. There is just something about them that can teach us a lot about life. One of my favorites is "Groundhog Day." The construct of this movie is that the main character Phil, played by Bill Murray, is trapped having to live the same day over and over again. This is what it can be like for IT when it comes to managing application performance. When a problem is detected in production, a "groundhog day" process is put into action to try and address that issue. Once the problem is addressed everyone resets and waits for the next problem to occur. In the end, the same process takes place over and over and the team never really makes an impact on the performance of an application.

In order to escape "Groundhog Day" there are some key practices that you can implement to build a team and transition the management of your applications' performance from firefighting to truly being proactive. Those key things are:

  • Getting some separation with the performance team
  • Picking the right skill set
  • Having the right tools in place

Each one of these practices changes the culture of application performance management (APM) from being reactive to becoming proactive. The concepts address some of the pitfalls that organizations face when trying to deal with the issue of performance problems.

To illustrate these key concepts let's look at how one of CompuwareAPM's customers, Raiffeisen Bank in Hungary, was able to change the culture of performance and escape its own "Groundhog Day." By implementing these key things the company was able to transition from variable performance in production with sporadic unplanned downtime to a more agile operation, deploying multiple releases a week with zero downtime.

Different Day - Same Results
Once Phil discovered that he was trapped in Punxatawney he attempted to escape his fate by trying to do the repeat his actions over and over. Similarly, most companies do the same thing when it comes to APM. I have discussed this scenario before. With Raiffeisen this was a common occurrence surrounding its portal application.

The teams at Raiffeisen were constantly fighting the same battles. It didn't matter whether it was during a peak traffic load or some random set of circumstances - the issue was always the same. The portal application's performance would degrade and the operations team had to spring into action. In some cases there was no indication that there was a performance problem other than the end users calling in stating they were having problems.

Poor Performance before the formation of a solid performance team

When a problem was detected, the operations team's process of performance management was reactive. First, the team would capture all the data relating to the performance of the portal. This included all the logs, dumps and hardware statistics associated with the portal. The data then had to be manually correlated and communicated to all the team members. This process was highly resource intensive both in time and manpower.

All of these measures still did not guarantee that the performance issues would be identified in time to rectify the issues. Without understanding the root cause of the performance bottlenecks, the team was left with few options. In a lot of cases the only way to address the issues in a timely fashion was with drastic measures. The application cluster had to be restarted. This was not an acceptable solution to these problems.

This is the "Groundhog Day" event that I am talking about. No matter what the team did the result was always the same. This is the constant fire drill that IT shops have been fighting about for years. The constant cycle of wait for the fire, find the fire, put out the fire as best as possible is an everyday occurrence for a lot of IT shops. Many APM tools that you might have implemented (legacy tools that can't copy any more and ‘point' solutions that only address one aspect of APM) in the past are usually only designed to cope with this cycle, not break it.

Continuous Improvement: The Only Way Out
Just as Phil was only able to escape his "day" after he began to continuously improve himself and others around him, The team at Raiffeisen realized the same thing for their own "Groundhog Day." They came to the understanding that what had to change was the way they were managing the performance of the portal application. The company made a conscious decision to fundamentally change the way it did APM as a team.

Get Some Separation
The first thing the team did was spin off a group and created their own team just to manage application performance. This is a key difference between performance initiatives that succeed and fail. The problem is that most companies assign the task of performance to one of the phases of the lifecycle. This isolates the impact that this team has over the entire lifecycle. For example, if this team is attached to testing then it has very little influence in the architecture of the application. The team also has no visibility other than what is provided to them by the tools currently in place.

Being separate gives the team a level of visibility so that its members can look at the whole process. This allows them to look for bottlenecks wherever their place in the lifecycle. This single act had a lasting impact at Raiffeisen. They had the ability to scrutinize the application at every level. When problems were discovered they had the visibility to make actionable recommendations.

The makeup of this team is as important as how it is positioned into the application lifecycle. When building a team to oversee the performance of the portal application Raiffeisen was looking for the right people. Each member was hand selected. This was not a task that was assigned to a single team or multiple groups of teams.

Selecting members who are very good detectives and subject matter experts is better than selecting the leads of the different application support groups. Picking people who have never worked on the application means they come with no potentially unhelpful prejudices and preconceptions: they're a blank slate. There are no thoughts of familiarity with the application that can lead to overlooking a problem.

Here, Raiffeisen was very specific about the criteria around how this team was to be built. The members of the team were selected based on their skills not their knowledge of the application. Each member of the team had no previous knowledge of the portal environment. Nor were they involved in any of the development or design phases. This was very important when it came to scrutinizing the application. The whole application was suspect until proven otherwise.

Right Tools for the Job
As discussed earlier, most performance tools that are in place are only good for fighting fires- not preventing them. To complete the transformation from reactive to proactive you need to get a solution that allows you to manage across the entire application lifecycle. Having the integrations and the ability to plug into the release process is crucial when selecting the right APM solution. Any solution that cannot do this is only upgrading the fire extinguisher.

When it came to Raiffeisen, it selected Compuware APM because of this. This was the turning point for dealing with the performance of the portal application. With this solution in place the newly formed team broke the fire-fighting cycle and started transforming the way Raiffeisen managed application performance. Teams easily shared data with each other, and they could clearly see the implications for actions needed to improve the performance of the application.

Fast Turn Around
The last "day" for Phil was after he was able to see how his life impacted those around him. With that he was able to escape Puxatawney and move on. With Raiffeisen that "day" ended two months after starting the project. The team was able to make a huge impact and change how the application was performing.

At the start of 2012 the performance team was created and Compuware APM was implemented. One month later the new application performance troubleshooting team had completed a full analysis of the portal application and created custom dashboards and views for all teams involved with the portal application: developers, testers, and operations managers. On top of that, developers started to implement changes into the application based on the findings from the performance troubleshooting team. In March of 2012 those changes were being tested and released into the production environment. Since the end of March there have been no unplanned outages. They have also been able to increase the number of updates to an almost daily occurrence. Additionally, the team was able to find problems over 30x faster than before and improved the login transaction from an average of 10 seconds to 3 seconds.

When the problem of performance is addressed across the lifecycle true change can be made. This is real lasting change in the way performance is managed. That is why managing performance is not just about having the newest tool in production. It is simply not enough. It just repeats the same cycle even though it may seem "easier" than the previous day. A company must realize that it is a combination of software, cultural change, and process that creates a lasting effect.

More Stories By Stephen Wilson

Stephen Wilson is a 15 year IT professional that currently holds the Subject Matter Expert role for Compuware APM within the Field Technology Sales organization. His role puts him in front of customers and their challenges on a daily basis. His background includes both development and operations. This kind of insight into the challenges that both developers face as well as those faced by the operational team allows him to be seen as a trusted advisor to his customers. His unique perspective into client needs and goals give creditability to the need for performance not just at one level but across the entire lifecycle.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

@CloudExpo Stories
"There's plenty of bandwidth out there but it's never in the right place. So what Cedexis does is uses data to work out the best pathways to get data from the origin to the person who wants to get it," explained Simon Jones, Evangelist and Head of Marketing at Cedexis, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
In his session at 21st Cloud Expo, Carl J. Levine, Senior Technical Evangelist for NS1, will objectively discuss how DNS is used to solve Digital Transformation challenges in large SaaS applications, CDNs, AdTech platforms, and other demanding use cases. Carl J. Levine is the Senior Technical Evangelist for NS1. A veteran of the Internet Infrastructure space, he has over a decade of experience with startups, networking protocols and Internet infrastructure, combined with the unique ability to it...
Agile has finally jumped the technology shark, expanding outside the software world. Enterprises are now increasingly adopting Agile practices across their organizations in order to successfully navigate the disruptive waters that threaten to drown them. In our quest for establishing change as a core competency in our organizations, this business-centric notion of Agile is an essential component of Agile Digital Transformation. In the years since the publication of the Agile Manifesto, the conn...
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
Enterprises are moving to the cloud faster than most of us in security expected. CIOs are going from 0 to 100 in cloud adoption and leaving security teams in the dust. Once cloud is part of an enterprise stack, it’s unclear who has responsibility for the protection of applications, services, and data. When cloud breaches occur, whether active compromise or a publicly accessible database, the blame must fall on both service providers and users. In his session at 21st Cloud Expo, Ben Johnson, C...
"We're developing a software that is based on the cloud environment and we are providing those services to corporations and the general public," explained Seungmin Kim, CEO/CTO of SM Systems Inc., in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Enterprises are adopting Kubernetes to accelerate the development and the delivery of cloud-native applications. However, sharing a Kubernetes cluster between members of the same team can be challenging. And, sharing clusters across multiple teams is even harder. Kubernetes offers several constructs to help implement segmentation and isolation. However, these primitives can be complex to understand and apply. As a result, it’s becoming common for enterprises to end up with several clusters. Thi...
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
Data scientists must access high-performance computing resources across a wide-area network. To achieve cloud-based HPC visualization, researchers must transfer datasets and visualization results efficiently. HPC clusters now compute GPU-accelerated visualization in the cloud cluster. To efficiently display results remotely, a high-performance, low-latency protocol transfers the display from the cluster to a remote desktop. Further, tools to easily mount remote datasets and efficiently transfer...
"Codigm is based on the cloud and we are here to explore marketing opportunities in America. Our mission is to make an ecosystem of the SW environment that anyone can understand, learn, teach, and develop the SW on the cloud," explained Sung Tae Ryu, CEO of Codigm, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
In his session at 21st Cloud Expo, James Henry, Co-CEO/CTO of Calgary Scientific Inc., introduced you to the challenges, solutions and benefits of training AI systems to solve visual problems with an emphasis on improving AIs with continuous training in the field. He explored applications in several industries and discussed technologies that allow the deployment of advanced visualization solutions to the cloud.
"Infoblox does DNS, DHCP and IP address management for not only enterprise networks but cloud networks as well. Customers are looking for a single platform that can extend not only in their private enterprise environment but private cloud, public cloud, tracking all the IP space and everything that is going on in that environment," explained Steve Salo, Principal Systems Engineer at Infoblox, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventio...
"We're focused on how to get some of the attributes that you would expect from an Amazon, Azure, Google, and doing that on-prem. We believe today that you can actually get those types of things done with certain architectures available in the market today," explained Steve Conner, VP of Sales at Cloudistics, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"NetApp is known as a data management leader but we do a lot more than just data management on-prem with the data centers of our customers. We're also big in the hybrid cloud," explained Wes Talbert, Principal Architect at NetApp, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Gemini is Yahoo’s native and search advertising platform. To ensure the quality of a complex distributed system that spans multiple products and components and across various desktop websites and mobile app and web experiences – both Yahoo owned and operated and third-party syndication (supply), with complex interaction with more than a billion users and numerous advertisers globally (demand) – it becomes imperative to automate a set of end-to-end tests 24x7 to detect bugs and regression. In th...
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
The question before companies today is not whether to become intelligent, it’s a question of how and how fast. The key is to adopt and deploy an intelligent application strategy while simultaneously preparing to scale that intelligence. In her session at 21st Cloud Expo, Sangeeta Chakraborty, Chief Customer Officer at Ayasdi, provided a tactical framework to become a truly intelligent enterprise, including how to identify the right applications for AI, how to build a Center of Excellence to oper...
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...