Click here to close now.

Welcome!

AJAX & REA Authors: Bernard Golden, Elizabeth White, Liz McMillan, Kevin Jackson, Hovhannes Avoyan

Related Topics: AJAX & REA

AJAX & REA: Article

The Log – So Important It Deserves a Book

A roll your own log management cookbook

I was very interested to see that Manning Publications have just announced the Early Access release of a new book called Unified Log Management. I looked through the table of contents and it was what I expected - a roll your own log management cookbook with a list of technologies that might help (Kinesis, Redshift, Kafka etc.). Presumably it will also include Google DataFlow by the time the book ships. So it's only really for those super-talented all-rounders with a lot of time on their hands or companies with the budget to hire a team of software developers and data scientists. Not really for the very busy DevOps or IT Ops crew with systems to keep available and the need for powerful yet easy to use tools to help them distill the hosepipe of data being generated.

What really peaked my interest was the blog post by the book's author, Alex Dean,  here. It turns out the "Unified" bit in the title was influenced by Jay Kreps article on the log. In my opinion, this is one of those articles that comes around once a generation! I don't think it's overstating the case to say that this captures all the most important lessons about big data, Internet of Things, massively scaled-out services in the cloud, growth hacking etc. It's up there with the 1988 paper that introduced the data warehouse concept and it does capture the future of enterprise architecture in a narrative that is refreshingly philosophical and erudite (at least for a tech blog post).

Enterprise Architecture The core ideas:

  • Capture every event that happens in your business - particularly every customer interaction
  • Store it multiple times because this stuff is gold
  • Pick the right tools to view this data from the perspectives that make sense for your business
  • Need better data - feed more into the log
  • Need a new perspective - add a new tool

The perspectives are visualizations of the output data. The tools are getting more specialized every day... time-series databases for metrics, complex event processing engines for security threats, distributed graph databases for social interactions, search indexing etc. If you know what you are looking for then use real-time pattern matching. If you don't know what you are looking for but want to be told when things start to look different then look for anomalies.

The log, according to Kreps, is at the heart of all these data transformations.

Stream data into the system as fast as your customers can generate it. On the other end you get insights out of the system. And in the middle you painstakingly, hand-craft your log transformation algorithms (with your team of superstar coders and data scientists).

If, on the other hand you have a business to run then why don't you feed data in here,here or here. And then see it transformed into insights like thisthis and this. Or alerts like these and these.

Logentries Webhook Integrations

We are busy working on new perspectives and visualizations. New tools and transformations. New insights and new alerts. The book is not out until Spring 2015 so we'll have a ton more features out before then.

Seeing as we started talking about unified logs - how about this for unified?

More Stories By Trevor Parsons

Trevor Parsons is Chief Scientist and Co-founder of Logentries. Trevor has over 10 years experience in enterprise software and, in particular, has specialized in developing enterprise monitoring and performance tools for distributed systems. He is also a research fellow at the Performance Engineering Lab Research Group and was formerly a Scientist at the IBM Center for Advanced Studies. Trevor holds a PhD from University College Dublin, Ireland.

@CloudExpo Stories
While DevOps most critically and famously fosters collaboration, communication, and integration through cultural change, culture is more of an output than an input. In order to actively drive cultural evolution, organizations must make substantial organizational and process changes, and adopt new technologies, to encourage a DevOps culture. Moderated by Andi Mann, panelists will discuss how to balance these three pillars of DevOps, where to focus attention (and resources), where organizations m...
Internet of Things (IoT) will be a hybrid ecosystem of diverse devices and sensors collaborating with operational and enterprise systems to create the next big application. In their session at @ThingsExpo, Bramh Gupta, founder and CEO of robomq.io, and Fred Yatzeck, principal architect leading product development at robomq.io, will discuss how choosing the right middleware and integration strategy from the get-go will enable IoT solution developers to adapt and grow with the industry, while at...
SYS-CON Events announced today that Column Technologies, a global technology solutions company, will exhibit at SYS-CON's DevOps Summit 2015 New York, which will take place June 9-11, 2015, at the Javits Center in New York City, NY. Established in 1998, Column Technologies is a leader in application performance and infrastructure management for commercial and federal markets. The company is headquartered in the United States, with a diverse and talented team of more than 350 employees around th...
The only place to be June 9-11 is Cloud Expo & @ThingsExpo 2015 East at the Javits Center in New York City. Join us there as delegates from all over the world come to listen to and engage with speakers & sponsors from the leading Cloud Computing, IoT & Big Data companies. Cloud Expo & @ThingsExpo are the leading events covering the booming market of Cloud Computing, IoT & Big Data for the enterprise. Speakers from all over the world will be hand-picked for their ability to explore the economic...
Deploying and operating cloud technology is difficult and falls outside the core competencies of most organizations. As a result, many on-premises private cloud installations falter, casting doubt on private cloud as a solution in general. Yet private clouds come with a unique set of benefits that continue to drive widespread interest. There must be a better way. This session will give participants a clear roadmap of private cloud implementation challenges and offer actionable ideas for...
Database apps on mobile devices shouldn't stop working when there's limited or no network connectivity. In his session at 16th Cloud Expo, Bradley Holt, a Developer Advocate for IBM Cloudant, will discuss how to bring data stored in a cloud database to the edge of the network (and back again), whenever an Internet connection is available. He will demonstrate techniques for replicating cloud databases with mobile devices in order to build offline-enabled mobile apps that can provide a better,...
SYS-CON Events announced today that the DevOps Institute has been named “Association Sponsor” of SYS-CON's DevOps Summit, which will take place on June 9–11, 2015, at the Javits Center in New York City, NY. The DevOps Institute provides enterprise level training and certification. Working with thought leaders from the DevOps community, the IT Service Management field and the IT training market, the DevOps Institute is setting the standard in quality for DevOps education and training.
Buzzword alert: Microservices and IoT at a DevOps conference? What could possibly go wrong? Join this panel of experts as they peel away the buzz and discuss the important architectural principles behind implementing IoT solutions for the enterprise. As remote IoT devices and sensors become increasingly intelligent, they become part of our distributed cloud environment, and we must architect and code accordingly. At the very least, you’ll have no problem filling in your buzzword bingo cards.
IoT is still a vague buzzword for many people. In his session at @ThingsExpo, Mike Kavis, Vice President & Principal Cloud Architect at Cloud Technology Partners, discussed the business value of IoT that goes far beyond the general public's perception that IoT is all about wearables and home consumer services. He also discussed how IoT is perceived by investors and how venture capitalist access this space. Other topics discussed were barriers to success, what is new, what is old, and what th...
SYS-CON Events announced today that CodeFutures, a leading supplier of database performance tools, has been named a “Sponsor” of SYS-CON's 16th International Cloud Expo®, which will take place on June 9–11, 2015, at the Javits Center in New York, NY. CodeFutures is an independent software vendor focused on providing tools that deliver database performance tools that increase productivity during database development and increase database performance and scalability during production.
SYS-CON Events announced today the IoT Bootcamp – Jumpstart Your IoT Strategy, being held June 9–10, 2015, in conjunction with 16th Cloud Expo and Internet of @ThingsExpo at the Javits Center in New York City. This is your chance to jumpstart your IoT strategy. Combined with real-world scenarios and use cases, the IoT Bootcamp is not just based on presentations but includes hands-on demos and walkthroughs. We will introduce you to a variety of Do-It-Yourself IoT platforms including Arduino, Ras...
Docker, Inc., has been included in the list of "Cool Vendors" in the April 21, 2015 report by Gartner, Inc, Cool Vendors in DevOps, 2015. In the report, Gartner notes, “As virtualization and cloud applications grow beyond being single images run on premise to being comprised of multiple tiers that can be run and scaled in a hybrid cloud, new approaches are needed to reduce complexity.” “Docker has gained great traction within the developer community for how efficient it makes the individual de...
SOA Software has changed its name to Akana. With roots in Web Services and SOA Governance, Akana has established itself as a leader in API Management and is expanding into cloud integration as an alternative to the traditional heavyweight enterprise service bus (ESB). The company recently announced that it achieved more than 90% year-over-year growth. As Akana, the company now addresses the evolution and diversification of SOA, unifying security, management, and DevOps across SOA, APIs, microser...
SYS-CON Events announced today that Creative Business Solutions will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Creative Business Solutions is the top stocking authorized HP Renew Distributor in the U.S. Based out of Long Island, NY, Creative Business Solutions offers a one-stop shop for a diverse range of products including Proliant, Blade and Industry Standard Servers, Networking, Server Options and...
Can call centers hang up the phones for good? Intuitive Solutions did. WebRTC enabled this contact center provider to eliminate antiquated telephony and desktop phone infrastructure with a pure web-based solution, allowing them to expand beyond brick-and-mortar confines to a home-based agent model. It also ensured scalability and better service for customers, including MUY! Companies, one of the country's largest franchise restaurant companies with 232 Pizza Hut locations. This is one example of...
Chuck Piluso will present a study of cloud adoption trends and the power and flexibility of IBM Power and Pureflex cloud solutions. Speaker Bio: Prior to Data Storage Corporation (DSC), Mr. Piluso founded North American Telecommunication Corporation, a facilities-based Competitive Local Exchange Carrier licensed by the Public Service Commission in 10 states, serving as the company's chairman and president from 1997 to 2000. Between 1990 and 1997, Mr. Piluso served as chairman & founder of ...
BlueBox bridge the chasm between development and infrastructure. Hosting providers are taking standardization and automation too far. For many app developers it does nothing but spawn mayhem and more work. They have to figure out how their creations live on a pre-fab infrastructure solution full of constraints. Operations-as-a-Service is what BlueBox does. BlueBox utilizes development tools such as OpenStack, EMC Razor, Opscode’s Chef and BlueBox's proprietary tools give the power to do the unor...
@ThingsExpo has been named the Top 5 Most Influential M2M Brand by Onalytica in the ‘Machine to Machine: Top 100 Influencers and Brands.' Onalytica analyzed the online debate on M2M by looking at over 85,000 tweets to provide the most influential individuals and brands that drive the discussion. According to Onalytica the "analysis showed a very engaged community with a lot of interactive tweets. The M2M discussion seems to be more fragmented and driven by some of the major brands present in the...
The WebRTC Summit 2015 New York, to be held June 9-11, 2015, at the Javits Center in New York, NY, announces that its Call for Papers is open. Topics include all aspects of improving IT delivery by eliminating waste through automated business models leveraging cloud technologies. WebRTC Summit is co-located with 16th International Cloud Expo, @ThingsExpo, Big Data Expo, and DevOps Summit.
SYS-CON Events announced today that Liaison Technologies, a leading provider of data management and integration cloud services and solutions, has been named "Silver Sponsor" of SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York, NY. Liaison Technologies is a recognized market leader in providing cloud-enabled data integration and data management solutions to break down complex information barriers, enabling enterprises to make sm...