Top big data tools used to store and analyze data
Contents |
[edit] Introduction
Big data is a phrase used for a collection of data sets so big and complex that it is difficult to process using traditional applications/tools. Due to the variety of information that it encompasses, big data consistently brings several challenges relating to its volume and complexity.
A recent survey claims that 80% of the data generated in the world are unstructured. One question is how these unstructured information can be structured, before we try to understand and capture the most important data. Another challenge is how we could store it. Listed below are the top tools utilised to store and analyse big data.
[edit] Apache Hadoop
Apache Hadoop is a java-based free software framework that can effectively store great deal of information in a cluster. This frame runs in parallel on a cluster and has an ability to enable us to process data across all nodes. Hadoop Distributed File System (HDFS) is the storage system of Hadoop which splits big information and distribute across several nodes in a cluster. This also replicates data in a bunch thus providing high availability.
[edit] Microsoft HDInsight
HDInsight utilises Windows Azure Blob storage as the default file system. This also provides high availability with reduced price.
[edit] NoSQL
While the traditional SQL can be effectively utilised to handle large quantity of structured data, we want NoSQL (Not Just SQL) to deal with unstructured data. NoSQL databases store unstructured information with no particular schema.
NoSQL gives better performance in storing massive number of data. There are lots of open-source NoSQL DBs available to analyse big data.
[edit] Hive
This supports SQL-like query option HiveSQL (HSQL) to get big data. This may be primarily used for its data-mining function.
[edit] Sqoop
This is a tool which connects Hadoop with various relational databases to transfer information. This can be effectively utilised to transport structured data to Hadoop or Hive.
[edit] PolyBase
This works on top of SQL Server 2012 Parallel Data Warehouse (PDW) and is used to get data stored in PDW. PDW is a data-warehousing appliance built for processing any quantity of relational data and provides an integration with Hadoop allowing the additional provision of non-relational information.
[edit] Big data in Excel
Lots of people are comfortable doing data analytics, therefore, the users may even connect data stored in Hadoop using Excel 2013. The Power View feature of Excel 2013 can be used to easily summarise the information. Similarly, Microsoft's HDInsight enables us to connect to big data stored in Azure Cloud using a power query option.
[edit] Presto
Facebook has developed and recently open-sourced its Query engine (SQL-on-Hadoop) called Presto which is built to manage petabytes of information. Unlike Hive, Presto doesn't depend on MapReduce technique and can quickly retrieve information.
[edit] Related articles on Designing Buildings Wiki
Featured articles and news
RTPI leader to become new CIOB Chief Executive Officer
Dr Victoria Hills MRTPI, FICE to take over after Caroline Gumble’s departure.
Social and affordable housing, a long term plan for delivery
The “Delivering a Decade of Renewal for Social and Affordable Housing” strategy sets out future path.
A change to adoptive architecture
Effects of global weather warming on architectural detailing, material choice and human interaction.
The proposed publicly owned and backed subsidiary of Homes England, to facilitate new homes.
How big is the problem and what can we do to mitigate the effects?
Overheating guidance and tools for building designers
A number of cool guides to help with the heat.
The UK's Modern Industrial Strategy: A 10 year plan
Previous consultation criticism, current key elements and general support with some persisting reservations.
Building Safety Regulator reforms
New roles, new staff and a new fast track service pave the way for a single construction regulator.
Architectural Technologist CPDs and Communications
CIAT CPD… and how you can do it!
Cooling centres and cool spaces
Managing extreme heat in cities by directing the public to places for heat stress relief and water sources.
Winter gardens: A brief history and warm variations
Extending the season with glass in different forms and terms.
Restoring Great Yarmouth's Winter Gardens
Transforming one of the least sustainable constructions imaginable.
Construction Skills Mission Board launch sector drive
Newly formed government and industry collaboration set strategy for recruiting an additional 100,000 construction workers a year.
New Architects Code comes into effect in September 2025
ARB Architects Code of Conduct and Practice available with ongoing consultation regarding guidance.
Welsh Skills Body (Medr) launches ambitious plan
The new skills body brings together funding and regulation of tertiary education and research for the devolved nation.
Paul Gandy FCIOB announced as next CIOB President
Former Tilbury Douglas CEO takes helm.
UK Infrastructure: A 10 Year Strategy. In brief with reactions
With the National Infrastructure and Service Transformation Authority (NISTA).