Welcome!

Machine Learning Authors: Liz McMillan, William Schmarzo, Madhavan Krishnan, VP, Cloud Solutions, Virtusa, Elizabeth White, Kevin Jackson

Related Topics: Agile Computing, Linux Containers, Open Source Cloud, Eclipse, Machine Learning , Apache

Agile Computing: Article

TraceView’s Oboe Symfony No. 1

The adoption of Symfony has already let it pull in some beautiful pieces of architecture

Dvořák’s Symphony No. 9 requires two oboes. Instrumenting Symfony only required one liboboe!

Here at AppNeta we’re still recovering from an immensely productive DrupalCon Portland 2013. Our partnership with Acquia and our continuing support for the contrib TraceView Drupal module made sponsoring the event a natural decision. We’re glad we made the long flight out West, as there was a lot of interest in TraceView. In fact, on the plane ride back to Boston I brought back not only three Voodoo Dozens but a stack of business cards two inches high. (I’ll leave it to you to guess which one ran out first!)

But for as much fun as I had exploring Portland’s unique culture, my favorite part of the con took place well before it. I spent a lot of time coding up improvements for TraceView’s Drupal support based on customer feedback. My personal goal was to submit enough patches to the TraceView Drupal module to warrant a version bump before DrupalCon, and thanks to the hard work of module maintainer Peter Drake, we were able to get there with room to spare. If you’re running Drupal 7, check out the changelog, and then go install it on your site!

The TraceView Drupal module now supports full-stack application tracing using drupal_http_request, including via the Apache Solr module.

I was able to code up more than just improvements in the months leading up to DrupalCon, though. If you stopped by our booth, you’ll have heard all about this, but we have a second important release to announce: we have a beta Symfony bundle and a (very!) preliminary port of our module to Drupal 8! While I had initially planned to just work on Drupal 6 and 7 improvements for TraceView, the announcement that Drupal 8′s SCOTCH and WSCII initiatives had resulted in the incorporation of Symfony components into Drupal 8 prompted me to take another look at what I’d lumped into ‘yet another PHP MVC framework’. It quickly became clear that I’d horribly misjudged Symfony. Drupal 8 hasn’t quite hit code freeze yet, but the adoption of Symfony has already let it pull in some beautiful pieces of architecture.

My favorite architectural decision in Symfony is its robust sub-request framework because of the degree of support that it lends to one of my favorite technologies, Edge Side Includes (ESI). Many large-scale sites leverage CDNs to deliver cached pages and static assets, but they don’t cope with logged-in users well because the content served to them is so heavily customized. ESI allows you to offload mostly cacheable pages to a CDN and only request the uncacheable portions of the page from the origin server, but it’s currently underutilized because most web frameworks can’t render just a portion of a page. In Symfony, though, any component of a page (such as a block or a sidebar) can be built as a sub-request for little additional developer effort. A year from now, large-scale Drupal 8 sites may entirely bypass common performance pain points by taking an ‘ESI Everywhere’ approach and leaving the CDN servers to do the work of assembling a dynamic page from a static cache entry.

Symfony had caught my interest, and my first instinct towards interesting software is to hack on it. That meant I had to create a Symfony site, so naturally, I chose Drupal 8 as a starting point. The codebase is still in flux, but it’s stable enough to install and it already makes use of several core Symfony components. All that I had to do to get an environment running was spin up an Amazon EC2 server, set up a LAMP stack and install TraceView (by far the fastest part of the process). The Drupal 8 installation required installing the trunk version of Drush, but after that I was only a `drush qd` away from a functioning Drupal 8 site.

I hit a wall fairly quickly, since like most Drupal developers I’d never tried writing Symfony code or using Composer to manage packages. Instead of getting frustrated, I took a step back and began to read articles about both Symfony in its own right and how it’s being leveraged to rewrite Drupal. I made heavy use of the Drupal core issue queue and the Symfony API documentation, but I also turned to blogs written by Drupal and Symfony developers. Of particular note, chx wrote an introduction to Symfony in two articles as well as an article that helped me fix my first Drupal 8 performance issue:

Every request to the Drupal 8 server triggered well over a hundred `file_get_contents` calls.

Before writing any integration code, I pulled up some Symfony traces because I was curious what they would look like. Even without code modification, TraceView picked up on Drupal 8 making a huge number of `file_get_contents` calls on each request. While individually fast, the calls often added up to more than 100ms; worse yet, that many disk reads has severe negative implications for scalability. At my last job, a sudden increase in traffic took down our entire distributed filesystem just by repeatedly fetching a single JavaScript file!

When I examined the `file_get_contents` calls in TraceView and realized they were all fetching module-related files, I immediately made the connection to Symfony’s dependency injection container (DIC) (one of its major design paradigms) thanks to another post by chx about how the DIC is stored on the filesystem. The filesystem calls were due to Drupal not finding the DIC on the filesystem and instead recompiling it using a dependency-checking mechanism similar to a package manager but made more complicated by the ability of classes to define additional compiler passes and other overrides. Like most problems I run into on Linux, I was able to trace it back to incorrect permissions:  because I had run Drush as a normal user, I had changed the permissions on the relevant directory and prevented the webserver from writing the compiled DIC so that it could be loaded on future requests. One `chown` later and my testbed site was running more than five times faster.

I kept an open mind to Symfony as I continued to read, but coming from the perspective of a Drupal developer, I was quite surprised to hear that the familiar hook-based event system has already been partially replaced with Symfony’s event listeners (bringing it more in line with some other PHP frameworks that we support). It’s new territory for me, but I can understand the rationale as there are some serious advantages to the switch. Unlike hooks, events don’t need to check every module on the site for an implementation; they can also have their propagation stopped if no further work is necessary. This means that developers will have much finer control over how modules respond to events, rather than having to use unusual hooks simply because they get called at the ‘right’ point in page loading (I’m looking at you, `hook_footer`!). Interestingly, a total switchover has been postponed until Drupal 9, which means that these overlapping systems will both remain in place for the foreseeable future.

The differences between Drupal hooks and Symfony event listeners were quite relevant to building out TraceView support, so I spent a lot of time reading about them. It’s there that I stumbled across another excellent addition to Drupal 8: the `terminate` kernel event! This import from Symfony is the replacement for the recently removed `hook_exit`, with the key difference being that it can be run after a response is returned to a browser. This lets expensive cache setting operations be performed in a way that won’t harm end user performance, emails be fired off only when the request is complete, or disk writes take place after an upload confirmation screen has been shown to the user.

It took about a month to get up to speed on Drupal 8 and Symfony to a degree where I felt ready to stop reading and start writing. That left me with about a month to go before DrupalCon, and in that time I was able to turn out not only the Drupal 7 module improvements but a really awesome DrupalCon booth demo!

The top tier of Drupalception included both our beta TraceView Drupal 8 module and our TraceView Symfony bundle.

I couldn’t think of any better way to show that we’re serious about our Drupal support than to demonstrate every version of Drupal we support at once, and so I built Drupalception: a Drupal 8 + Symfony AWS instance using cURL (via Guzzle) to call into our Drupal 7 demo site, which in turn uses drupal_http_request to call into our Drupal 6 demo site. TraceView catches the entire stack as a single trace even when blocking calls are nested fifteen layers deep!

But wait, aren’t I skipping a step – how did I get from reading a bunch of blog articles about Drupal 8 in Symfony to a functioning Drupal 8 module and Symfony bundle? Well, that’s a story in its own right – and for now, one that you’ll have to wait for another blog post for! Until next time, sign up for our 14 day trial and try out firsthand our support for Drupal 6, 7, or even 8 (or any other stack we support, for that matter).

Related Articles

Drupal Performance Profiling With Tracelytics

Headnet Improves Drupal Performance and Reliability With TraceView

Case Study: Penton Media Improves its Drupal Application Performance

More Stories By James Meickle

James started as a hobbyist web developer, even though his academic background is in social psychology and political science. Lately his interests as a professional Drupal developer have migrated towards performance, security, and automation. His favorite languages is Python, his favorite editor is Sublime, and his favorite game is Dwarf Fortress.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@CloudExpo Stories
"We work around really protecting the confidentiality of information, and by doing so we've developed implementations of encryption through a patented process that is known as superencipherment," explained Richard Blech, CEO of Secure Channels Inc., in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
High-velocity engineering teams are applying not only continuous delivery processes, but also lessons in experimentation from established leaders like Amazon, Netflix, and Facebook. These companies have made experimentation a foundation for their release processes, allowing them to try out major feature releases and redesigns within smaller groups before making them broadly available. In his session at 21st Cloud Expo, Brian Lucas, Senior Staff Engineer at Optimizely, discussed how by using ne...
"Infoblox does DNS, DHCP and IP address management for not only enterprise networks but cloud networks as well. Customers are looking for a single platform that can extend not only in their private enterprise environment but private cloud, public cloud, tracking all the IP space and everything that is going on in that environment," explained Steve Salo, Principal Systems Engineer at Infoblox, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventio...
Agile has finally jumped the technology shark, expanding outside the software world. Enterprises are now increasingly adopting Agile practices across their organizations in order to successfully navigate the disruptive waters that threaten to drown them. In our quest for establishing change as a core competency in our organizations, this business-centric notion of Agile is an essential component of Agile Digital Transformation. In the years since the publication of the Agile Manifesto, the conn...
While some developers care passionately about how data centers and clouds are architected, for most, it is only the end result that matters. To the majority of companies, technology exists to solve a business problem, and only delivers value when it is solving that problem. 2017 brings the mainstream adoption of containers for production workloads. In his session at 21st Cloud Expo, Ben McCormack, VP of Operations at Evernote, discussed how data centers of the future will be managed, how the p...
"Cloud Academy is an enterprise training platform for the cloud, specifically public clouds. We offer guided learning experiences on AWS, Azure, Google Cloud and all the surrounding methodologies and technologies that you need to know and your teams need to know in order to leverage the full benefits of the cloud," explained Alex Brower, VP of Marketing at Cloud Academy, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clar...
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
In his session at 21st Cloud Expo, James Henry, Co-CEO/CTO of Calgary Scientific Inc., introduced you to the challenges, solutions and benefits of training AI systems to solve visual problems with an emphasis on improving AIs with continuous training in the field. He explored applications in several industries and discussed technologies that allow the deployment of advanced visualization solutions to the cloud.
"There's plenty of bandwidth out there but it's never in the right place. So what Cedexis does is uses data to work out the best pathways to get data from the origin to the person who wants to get it," explained Simon Jones, Evangelist and Head of Marketing at Cedexis, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"Codigm is based on the cloud and we are here to explore marketing opportunities in America. Our mission is to make an ecosystem of the SW environment that anyone can understand, learn, teach, and develop the SW on the cloud," explained Sung Tae Ryu, CEO of Codigm, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"We're developing a software that is based on the cloud environment and we are providing those services to corporations and the general public," explained Seungmin Kim, CEO/CTO of SM Systems Inc., in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"We're focused on how to get some of the attributes that you would expect from an Amazon, Azure, Google, and doing that on-prem. We believe today that you can actually get those types of things done with certain architectures available in the market today," explained Steve Conner, VP of Sales at Cloudistics, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
The question before companies today is not whether to become intelligent, it’s a question of how and how fast. The key is to adopt and deploy an intelligent application strategy while simultaneously preparing to scale that intelligence. In her session at 21st Cloud Expo, Sangeeta Chakraborty, Chief Customer Officer at Ayasdi, provided a tactical framework to become a truly intelligent enterprise, including how to identify the right applications for AI, how to build a Center of Excellence to oper...
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
DevOps promotes continuous improvement through a culture of collaboration. But in real terms, how do you: Integrate activities across diverse teams and services? Make objective decisions with system-wide visibility? Use feedback loops to enable learning and improvement? With technology insights and real-world examples, in his general session at @DevOpsSummit, at 21st Cloud Expo, Andi Mann, Chief Technology Advocate at Splunk, explored how leading organizations use data-driven DevOps to close th...
Enterprises are moving to the cloud faster than most of us in security expected. CIOs are going from 0 to 100 in cloud adoption and leaving security teams in the dust. Once cloud is part of an enterprise stack, it’s unclear who has responsibility for the protection of applications, services, and data. When cloud breaches occur, whether active compromise or a publicly accessible database, the blame must fall on both service providers and users. In his session at 21st Cloud Expo, Ben Johnson, C...