Welcome!

Machine Learning Authors: Zakia Bouachraoui, William Schmarzo, Yeshim Deniz, Elizabeth White, Pat Romanski

Related Topics: Machine Learning , Recurring Revenue, Server Monitoring, @CloudExpo, @DXWorldExpo, @ThingsExpo

Machine Learning : Blog Post

Data Lake and Data Refinery | @ThingsExpo #BigData #IoT #M2M #API #InternetOfThings

The data lake strategy is part of a greater movement toward data liberalization

Data Lake and Data Refinery – Gartner Controversy!

Much discussion has been going on the new phrase called Data Lake. Gartner wrote a report on the ‘Data Lake’ fallacy, saying to be careful about ‘data lake’ or ‘data swamp’. Then Andrew Oliver wrote in the InfoWorld these beginning words, “For $200, Gartner tells you ‘data lakes’ are bad and advises you to try real hard, plan far in advance, and get governance correct”. Wow, what an insight!

During my days at IBM and Oracle, Gartner wanted to get time on my calendar to talk about database futures. Then afterwards, I realized that I paid significant fee to attain the Gartner conference to hear back what I had told them. Good business of information gathering and selling back. Without meaning any disrespect, many analysts like to create controversial statements to stay relevant. Here is such a case with Gartner.

The concept of a ‘data lake’ was coined by James Dixon of Pentaho Corp. and this is what he said – If you think of a datamart as a store of bottled water – cleansed and packaged and structured for easy consumption – the data lake is a large body of water in a more natural state. The contents of the data lake stream in from a source to fill the lake, and various users of the lake can come to examine, dive in, or take samples. Think of a data lake as an unstructured data warehouse, a place where you pull in all of your different sources into one large “pool” of data. In contrast to a data mart, a data lake won’t “wash” the data or try to structure it or limit the use cases. Sure, you should have some use cases in mind, but the architecture of a data lake is simple: a Hadoop File System (HDFS) with lots of directories and files on it.

The data lake strategy is part of a greater movement toward data liberalization. Given the exponential growth of data (specially with IoT and myriads of sensors), there is need for storing data in its native format for further analysis. Of course you can drown in a data lake! But that’s why you build safety nets like security procedures (for example, access is allowed only via Knox), documentation (what goes where in what directory and what roles you need to find it), and  governance.

Without offering any concrete alternative, Gartner seems to say that a new layer (call it data refinery if you like) is needed to make sense of this ‘raw’ data, thus heading back to the ETL days of data warehousing. Gartner loves to scare clients (so that they seek help for a fee) on new technology and would want everyone to stay with the classic data warehousing business. This is not in line with the Big Data movement which involves some risk, as always with any new technology. 

Read the original blog entry...

More Stories By Jnan Dash

Jnan Dash is Senior Advisor at EZShield Inc., Advisor at ScaleDB and Board Member at Compassites Software Solutions. He has lived in Silicon Valley since 1979. Formerly he was the Chief Strategy Officer (Consulting) at Curl Inc., before which he spent ten years at Oracle Corporation and was the Group Vice President, Systems Architecture and Technology till 2002. He was responsible for setting Oracle's core database and application server product directions and interacted with customers worldwide in translating future needs to product plans. Before that he spent 16 years at IBM. He blogs at http://jnandash.ulitzer.com.

CloudEXPO Stories
Darktrace is the world's leading AI company for cyber security. Created by mathematicians from the University of Cambridge, Darktrace's Enterprise Immune System is the first non-consumer application of machine learning to work at scale, across all network types, from physical, virtualized, and cloud, through to IoT and industrial control systems. Installed as a self-configuring cyber defense platform, Darktrace continuously learns what is ‘normal' for all devices and users, updating its understanding as the environment changes.
Industry after industry is under siege as companies embrace digital transformation (DX) to disrupt existing business models and disintermediate their competitor’s customer relationships. But what do we mean by “Digital Transformation”? The coupling of granular, real-time data (e.g., smartphones, connected devices, smart appliances, wearables, mobile commerce, video surveillance) with modern technologies (e.g., cloud native apps, big data architectures, hyper-converged technologies, artificial intelligence, blockchain) to enhance products, processes, and business-decision making with customer, product and operational insights.
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throughout enterprises of all sizes.
CloudEXPO has been the M&A capital for Cloud companies for more than a decade with memorable acquisition news stories which came out of CloudEXPO expo floor. DevOpsSUMMIT New York faculty member Greg Bledsoe shared his views on IBM's Red Hat acquisition live from NASDAQ floor. Acquisition news was announced during CloudEXPO New York which took place November 12-13, 2019 in New York City.
OpsRamp is an enterprise IT operation platform provided by US-based OpsRamp, Inc. It provides SaaS services through support for increasingly complex cloud and hybrid computing environments from system operation to service management. The OpsRamp platform is a SaaS-based, multi-tenant solution that enables enterprise IT organizations and cloud service providers like JBS the flexibility and control they need to manage and monitor today's hybrid, multi-cloud infrastructure, applications, and workloads, including Microsoft Azure. We are excited to partner with JBS and look forward to a long and successful relationship.