Welcome!

Machine Learning Authors: Liz McMillan, Ed Featherston, Destiny Bertucci, Elizabeth White, Dan Blacharski

Related Topics: Agile Computing, Linux Containers, Open Source Cloud, Eclipse, Machine Learning , Apache

Agile Computing: Article

TraceView’s Oboe Symfony No. 1

The adoption of Symfony has already let it pull in some beautiful pieces of architecture

Dvořák’s Symphony No. 9 requires two oboes. Instrumenting Symfony only required one liboboe!

Here at AppNeta we’re still recovering from an immensely productive DrupalCon Portland 2013. Our partnership with Acquia and our continuing support for the contrib TraceView Drupal module made sponsoring the event a natural decision. We’re glad we made the long flight out West, as there was a lot of interest in TraceView. In fact, on the plane ride back to Boston I brought back not only three Voodoo Dozens but a stack of business cards two inches high. (I’ll leave it to you to guess which one ran out first!)

But for as much fun as I had exploring Portland’s unique culture, my favorite part of the con took place well before it. I spent a lot of time coding up improvements for TraceView’s Drupal support based on customer feedback. My personal goal was to submit enough patches to the TraceView Drupal module to warrant a version bump before DrupalCon, and thanks to the hard work of module maintainer Peter Drake, we were able to get there with room to spare. If you’re running Drupal 7, check out the changelog, and then go install it on your site!

The TraceView Drupal module now supports full-stack application tracing using drupal_http_request, including via the Apache Solr module.

I was able to code up more than just improvements in the months leading up to DrupalCon, though. If you stopped by our booth, you’ll have heard all about this, but we have a second important release to announce: we have a beta Symfony bundle and a (very!) preliminary port of our module to Drupal 8! While I had initially planned to just work on Drupal 6 and 7 improvements for TraceView, the announcement that Drupal 8′s SCOTCH and WSCII initiatives had resulted in the incorporation of Symfony components into Drupal 8 prompted me to take another look at what I’d lumped into ‘yet another PHP MVC framework’. It quickly became clear that I’d horribly misjudged Symfony. Drupal 8 hasn’t quite hit code freeze yet, but the adoption of Symfony has already let it pull in some beautiful pieces of architecture.

My favorite architectural decision in Symfony is its robust sub-request framework because of the degree of support that it lends to one of my favorite technologies, Edge Side Includes (ESI). Many large-scale sites leverage CDNs to deliver cached pages and static assets, but they don’t cope with logged-in users well because the content served to them is so heavily customized. ESI allows you to offload mostly cacheable pages to a CDN and only request the uncacheable portions of the page from the origin server, but it’s currently underutilized because most web frameworks can’t render just a portion of a page. In Symfony, though, any component of a page (such as a block or a sidebar) can be built as a sub-request for little additional developer effort. A year from now, large-scale Drupal 8 sites may entirely bypass common performance pain points by taking an ‘ESI Everywhere’ approach and leaving the CDN servers to do the work of assembling a dynamic page from a static cache entry.

Symfony had caught my interest, and my first instinct towards interesting software is to hack on it. That meant I had to create a Symfony site, so naturally, I chose Drupal 8 as a starting point. The codebase is still in flux, but it’s stable enough to install and it already makes use of several core Symfony components. All that I had to do to get an environment running was spin up an Amazon EC2 server, set up a LAMP stack and install TraceView (by far the fastest part of the process). The Drupal 8 installation required installing the trunk version of Drush, but after that I was only a `drush qd` away from a functioning Drupal 8 site.

I hit a wall fairly quickly, since like most Drupal developers I’d never tried writing Symfony code or using Composer to manage packages. Instead of getting frustrated, I took a step back and began to read articles about both Symfony in its own right and how it’s being leveraged to rewrite Drupal. I made heavy use of the Drupal core issue queue and the Symfony API documentation, but I also turned to blogs written by Drupal and Symfony developers. Of particular note, chx wrote an introduction to Symfony in two articles as well as an article that helped me fix my first Drupal 8 performance issue:

Every request to the Drupal 8 server triggered well over a hundred `file_get_contents` calls.

Before writing any integration code, I pulled up some Symfony traces because I was curious what they would look like. Even without code modification, TraceView picked up on Drupal 8 making a huge number of `file_get_contents` calls on each request. While individually fast, the calls often added up to more than 100ms; worse yet, that many disk reads has severe negative implications for scalability. At my last job, a sudden increase in traffic took down our entire distributed filesystem just by repeatedly fetching a single JavaScript file!

When I examined the `file_get_contents` calls in TraceView and realized they were all fetching module-related files, I immediately made the connection to Symfony’s dependency injection container (DIC) (one of its major design paradigms) thanks to another post by chx about how the DIC is stored on the filesystem. The filesystem calls were due to Drupal not finding the DIC on the filesystem and instead recompiling it using a dependency-checking mechanism similar to a package manager but made more complicated by the ability of classes to define additional compiler passes and other overrides. Like most problems I run into on Linux, I was able to trace it back to incorrect permissions:  because I had run Drush as a normal user, I had changed the permissions on the relevant directory and prevented the webserver from writing the compiled DIC so that it could be loaded on future requests. One `chown` later and my testbed site was running more than five times faster.

I kept an open mind to Symfony as I continued to read, but coming from the perspective of a Drupal developer, I was quite surprised to hear that the familiar hook-based event system has already been partially replaced with Symfony’s event listeners (bringing it more in line with some other PHP frameworks that we support). It’s new territory for me, but I can understand the rationale as there are some serious advantages to the switch. Unlike hooks, events don’t need to check every module on the site for an implementation; they can also have their propagation stopped if no further work is necessary. This means that developers will have much finer control over how modules respond to events, rather than having to use unusual hooks simply because they get called at the ‘right’ point in page loading (I’m looking at you, `hook_footer`!). Interestingly, a total switchover has been postponed until Drupal 9, which means that these overlapping systems will both remain in place for the foreseeable future.

The differences between Drupal hooks and Symfony event listeners were quite relevant to building out TraceView support, so I spent a lot of time reading about them. It’s there that I stumbled across another excellent addition to Drupal 8: the `terminate` kernel event! This import from Symfony is the replacement for the recently removed `hook_exit`, with the key difference being that it can be run after a response is returned to a browser. This lets expensive cache setting operations be performed in a way that won’t harm end user performance, emails be fired off only when the request is complete, or disk writes take place after an upload confirmation screen has been shown to the user.

It took about a month to get up to speed on Drupal 8 and Symfony to a degree where I felt ready to stop reading and start writing. That left me with about a month to go before DrupalCon, and in that time I was able to turn out not only the Drupal 7 module improvements but a really awesome DrupalCon booth demo!

The top tier of Drupalception included both our beta TraceView Drupal 8 module and our TraceView Symfony bundle.

I couldn’t think of any better way to show that we’re serious about our Drupal support than to demonstrate every version of Drupal we support at once, and so I built Drupalception: a Drupal 8 + Symfony AWS instance using cURL (via Guzzle) to call into our Drupal 7 demo site, which in turn uses drupal_http_request to call into our Drupal 6 demo site. TraceView catches the entire stack as a single trace even when blocking calls are nested fifteen layers deep!

But wait, aren’t I skipping a step – how did I get from reading a bunch of blog articles about Drupal 8 in Symfony to a functioning Drupal 8 module and Symfony bundle? Well, that’s a story in its own right – and for now, one that you’ll have to wait for another blog post for! Until next time, sign up for our 14 day trial and try out firsthand our support for Drupal 6, 7, or even 8 (or any other stack we support, for that matter).

Related Articles

Drupal Performance Profiling With Tracelytics

Headnet Improves Drupal Performance and Reliability With TraceView

Case Study: Penton Media Improves its Drupal Application Performance

More Stories By James Meickle

James started as a hobbyist web developer, even though his academic background is in social psychology and political science. Lately his interests as a professional Drupal developer have migrated towards performance, security, and automation. His favorite languages is Python, his favorite editor is Sublime, and his favorite game is Dwarf Fortress.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@CloudExpo Stories
"Storpool does only block-level storage so we do one thing extremely well. The growth in data is what drives the move to software-defined technologies in general and software-defined storage," explained Boyan Ivanov, CEO and co-founder at StorPool, in this SYS-CON.tv interview at 16th Cloud Expo, held June 9-11, 2015, at the Javits Center in New York City.
You know you need the cloud, but you’re hesitant to simply dump everything at Amazon since you know that not all workloads are suitable for cloud. You know that you want the kind of ease of use and scalability that you get with public cloud, but your applications are architected in a way that makes the public cloud a non-starter. You’re looking at private cloud solutions based on hyperconverged infrastructure, but you’re concerned with the limits inherent in those technologies.
Is advanced scheduling in Kubernetes achievable?Yes, however, how do you properly accommodate every real-life scenario that a Kubernetes user might encounter? How do you leverage advanced scheduling techniques to shape and describe each scenario in easy-to-use rules and configurations? In his session at @DevOpsSummit at 21st Cloud Expo, Oleg Chunikhin, CTO at Kublr, answered these questions and demonstrated techniques for implementing advanced scheduling. For example, using spot instances and co...
As Marc Andreessen says software is eating the world. Everything is rapidly moving toward being software-defined – from our phones and cars through our washing machines to the datacenter. However, there are larger challenges when implementing software defined on a larger scale - when building software defined infrastructure. In his session at 16th Cloud Expo, Boyan Ivanov, CEO of StorPool, provided some practical insights on what, how and why when implementing "software-defined" in the datacent...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
The cloud era has reached the stage where it is no longer a question of whether a company should migrate, but when. Enterprises have embraced the outsourcing of where their various applications are stored and who manages them, saving significant investment along the way. Plus, the cloud has become a defining competitive edge. Companies that fail to successfully adapt risk failure. The media, of course, continues to extol the virtues of the cloud, including how easy it is to get there. Migrating...
The use of containers by developers -- and now increasingly IT operators -- has grown from infatuation to deep and abiding love. But as with any long-term affair, the honeymoon soon leads to needing to live well together ... and maybe even getting some relationship help along the way. And so it goes with container orchestration and automation solutions, which are rapidly emerging as the means to maintain the bliss between rapid container adoption and broad container use among multiple cloud host...
Blockchain is a shared, secure record of exchange that establishes trust, accountability and transparency across business networks. Supported by the Linux Foundation's open source, open-standards based Hyperledger Project, Blockchain has the potential to improve regulatory compliance, reduce cost as well as advance trade. Are you curious about how Blockchain is built for business? In her session at 21st Cloud Expo, René Bostic, Technical VP of the IBM Cloud Unit in North America, discussed the b...
Imagine if you will, a retail floor so densely packed with sensors that they can pick up the movements of insects scurrying across a store aisle. Or a component of a piece of factory equipment so well-instrumented that its digital twin provides resolution down to the micrometer.
The need for greater agility and scalability necessitated the digital transformation in the form of following equation: monolithic to microservices to serverless architecture (FaaS). To keep up with the cut-throat competition, the organisations need to update their technology stack to make software development their differentiating factor. Thus microservices architecture emerged as a potential method to provide development teams with greater flexibility and other advantages, such as the abili...
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settle...
Blockchain. A day doesn’t seem to go by without seeing articles and discussions about the technology. According to PwC executive Seamus Cushley, approximately $1.4B has been invested in blockchain just last year. In Gartner’s recent hype cycle for emerging technologies, blockchain is approaching the peak. It is considered by Gartner as one of the ‘Key platform-enabling technologies to track.’ While there is a lot of ‘hype vs reality’ discussions going on, there is no arguing that blockchain is b...
Product connectivity goes hand and hand these days with increased use of personal data. New IoT devices are becoming more personalized than ever before. In his session at 22nd Cloud Expo | DXWorld Expo, Nicolas Fierro, CEO of MIMIR Blockchain Solutions, will discuss how in order to protect your data and privacy, IoT applications need to embrace Blockchain technology for a new level of product security never before seen - or needed.
As DevOps methodologies expand their reach across the enterprise, organizations face the daunting challenge of adapting related cloud strategies to ensure optimal alignment, from managing complexity to ensuring proper governance. How can culture, automation, legacy apps and even budget be reexamined to enable this ongoing shift within the modern software factory? In her Day 2 Keynote at @DevOpsSummit at 21st Cloud Expo, Aruna Ravichandran, VP, DevOps Solutions Marketing, CA Technologies, was jo...
Leading companies, from the Global Fortune 500 to the smallest companies, are adopting hybrid cloud as the path to business advantage. Hybrid cloud depends on cloud services and on-premises infrastructure working in unison. Successful implementations require new levels of data mobility, enabled by an automated and seamless flow across on-premises and cloud resources. In his general session at 21st Cloud Expo, Greg Tevis, an IBM Storage Software Technical Strategist and Customer Solution Architec...
Nordstrom is transforming the way that they do business and the cloud is the key to enabling speed and hyper personalized customer experiences. In his session at 21st Cloud Expo, Ken Schow, VP of Engineering at Nordstrom, discussed some of the key learnings and common pitfalls of large enterprises moving to the cloud. This includes strategies around choosing a cloud provider(s), architecture, and lessons learned. In addition, he covered some of the best practices for structured team migration an...
In his general session at 21st Cloud Expo, Greg Dumas, Calligo’s Vice President and G.M. of US operations, discussed the new Global Data Protection Regulation and how Calligo can help business stay compliant in digitally globalized world. Greg Dumas is Calligo's Vice President and G.M. of US operations. Calligo is an established service provider that provides an innovative platform for trusted cloud solutions. Calligo’s customers are typically most concerned about GDPR compliance, application p...
Coca-Cola’s Google powered digital signage system lays the groundwork for a more valuable connection between Coke and its customers. Digital signs pair software with high-resolution displays so that a message can be changed instantly based on what the operator wants to communicate or sell. In their Day 3 Keynote at 21st Cloud Expo, Greg Chambers, Global Group Director, Digital Innovation, Coca-Cola, and Vidya Nagarajan, a Senior Product Manager at Google, discussed how from store operations and ...
In his session at 21st Cloud Expo, Raju Shreewastava, founder of Big Data Trunk, provided a fun and simple way to introduce Machine Leaning to anyone and everyone. He solved a machine learning problem and demonstrated an easy way to be able to do machine learning without even coding. Raju Shreewastava is the founder of Big Data Trunk (www.BigDataTrunk.com), a Big Data Training and consulting firm with offices in the United States. He previously led the data warehouse/business intelligence and B...
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...