After 10 years, the world will move into a new era – the era of big data. Instead of a weather widget on the smartphone screen, he himself will tell you what is best to wear. At breakfast, the phone will show the road along which you will quickly get to work and when you need to leave.
Under the influence of Big Data, everything that a person does not touch will change. We will figure out what it is, and also consider the real application and prospects of the technology.
What is Big data?
Big data is an information processing technology that surpasses hundreds of terabytes and grows exponentially over time.
Such data is so large and complex that none of the traditional data management tools can store or efficiently process it. A person is not able to analyze this volume. For this, special algorithms have been developed that, after analyzing big data, give a person understandable results.
Big Data includes petabytes (1024 terabytes) or exabytes (1024 petabytes) of information that make up billions or trillions of records of millions of people and all from different sources (Internet, sales, contact center, social networks, mobile devices). As a rule, information is poorly structured and often incomplete and inaccessible.
How does Big-Data technology work?
Users of the social network Facebook upload photos, videos and perform actions every day for hundreds of terabytes. No matter how many people participate in the development, they will not cope with the constant flow of information. In order to further develop the service and make sites more comfortable – introduce smart content recommendations, display ads relevant to the user, hundreds of thousands of terabytes are passed through the algorithm and receive structured and understandable information.
Comparing a huge amount of information, it finds the relationship. These relationships are likely to predict the future. To find and analyze a person helps artificial intelligence.
The neural network scans thousands of photos, videos, comments – the very hundreds of terabytes of big data and gives the result: how many satisfied customers leave the store, whether there will be a traffic jam in the coming hours, what discussions are popular on the social network and much more.
Methods of working with big data:
- Machine learning
- Mood analysis
- Social network analysis
- Learning Rules Association
- Classification Tree Analysis
- Genetic Algorithms
- Regression analysis
Machine learning
You look at the news feed, like Instagram posts, and the algorithm examines your content and recommends similar ones. Artificial intelligence learns without explicit programming and focuses on forecasting based on well-known properties extracted from sets of “training data”.
Machine learning helps :
- Distinguish between spam and non-spam email
- Explore user preferences and give recommendations
- Identify the best content to attract potential customers.
- Determine the probability of winning a case and set legal fees
Mood analysis
Mood analysis helps :
- Improve hotel chain service by analyzing guest comments
- Customize incentives and services to meet customer needs
- Determine from opinions on the social network what customers are thinking.
Social Network Analysis
Social network analysis was first used in the telecommunications industry. The method is used by sociologists to analyze relationships between people in many fields and commercial activities.
This analysis is used to :
- See how people from different population groups form relationships with outsiders
- Find out the importance and influence of a specific person in a group
- Find the minimum number of direct connections to connect two people
- Understand the social structure of the customer base
Learning Association Rules
People who don’t buy alcohol take juices more often than lovers of strong drinks?
Studying association rules is a method for discovering interesting relationships between variables in large databases. For the first time, it was used by large supermarket chains to discover interesting relationships between products using information from supermarket outlet systems (POS).
Using association rules :
- Place products closer to each other to increase sales
- Extract website visitor information from web server logs
- Analyze biological data
- Track system logs to detect intruders
- Determine if tea shoppers take sodas more often
Classification Tree Analysis
Statistical classification defines the categories to which the new observation belongs.
Statistical classification is used for :
- Automatic assignment of documents to categories
- Classification of organisms into groups
- Developing profiles of students taking online courses
Genetic Algorithms
Genetic algorithms are inspired by how evolution works, that is, through mechanisms such as inheritance, mutation, and natural selection.
Genetic algorithms are used for :
- Scheduling doctors for emergency departments in hospitals
- Calculation of optimal materials for the development of fuel-efficient cars
- Creating “artificially creative” content such as puns and jokes
Regression analysis
How does a person’s age affect the type of car he buys?
At a basic level, regression analysis involves manipulating some independent variable (like background music) to see how it affects the dependent variable (time spent in the store).
Regression analysis is used to determine:
- Customer satisfaction levels
- How the weather forecast for the previous day affects the number of calls to support
- How the area and size of houses affect the price of housing
Data Mining – how Big Date is collected and processed
Uploading big data to a traditional relational database for analysis takes a lot of time and money. For this reason, special approaches have appeared for the collection and analysis of information. To obtain and then retrieve the information, combine and place it in a “data lake”. From there, artificial intelligence programs using complex algorithms look for repeating patterns.
Storage and processing takes place with the following tools:
- Apache HADOOP is a packet-oriented data processing system. The system stores and tracks information on several machines and scales to several thousand servers.
- HPPC is an open source platform developed by LexisNexis Risk Solutions. HPPC is known as the Data Analytics Supercomputer (DAS), which supports data processing both in batch mode and in real time. The system uses supercomputers and clusters from conventional computers.
- Storm – processes information in real time. Uses the open source Eclipse Public License.
The real application of Big Data
The fastest growth in spending on big data technology occurs in the banking, healthcare, insurance, securities and investment services, as well as in telecommunications. Three of these industries are in the financial sector, which has many useful options for Big Data analysis: fraud detection, risk management, and customer service optimization.
Banks and credit card companies use big data to identify patterns that indicate criminal activity. Because of this, some analysts believe that big data can benefit cryptocurrency . Algorithms will be able to detect fraud and illegal activities in the crypto industry.
Thanks to cryptocurrencies such as Bitcoin and Ethereum, the blockchain can actually support any type of digitized information. It can be used in the Big Data field, especially to improve the security or quality of information.
For example, a hospital can use it to ensure the safety, relevance of patient data and to fully preserve their quality. By placing health databases on the blockchain, the hospital provides all its employees with access to a single, unchanging source of information.
Just as people associate cryptocurrency with volatility, they often associate big data with the ability to sift through large amounts of information. Big Data helps track trends. A lot of factors influence the price and big data algorithms will take this into account and then provide a solution.
Prospects for Using Big Date
Blockchain and Big Data are two evolving and complementary technologies. Since 2016, blockchain is often discussed in the media. This is a cryptographically secure distributed database technology for storing and transmitting information. Protecting private and confidential information is an urgent and future big data problem that the blockchain can solve.
Almost every industry has begun investing in Big Data analytics, but some are investing more than others. According to IDC, they spend more on banking, discrete manufacturing, process manufacturing and professional services. According to Wikibon research, revenue from sales of programs and services on the world market in 2018 amounted to $ 42 billion, and in 2027 it will overcome the mark of $ 100 billion.
Neimeth estimates that the blockchain will account for up to 20% of the total big data market by 2030, generating up to $ 100 billion in annual revenue. This exceeds the profit of PayPal, Visa and Mastercard combined.
Big Data analytics will be important for tracking transactions and will allow companies using the blockchain to identify hidden schemes and find out who they interact with on the blockchain.
Big data market in Russia
The whole world, including Russia, uses Big Data technology in banking, communications and retail. Experts believe that in the future technology will be used by the transport industry, the oil and gas and food industries, as well as energy.
IDC analysts have recognized Russia as the largest regional BDA market. According to estimates this year, revenue will approach $ 1.4 billion and will account for 40% of total investment in the big data sector and business intelligence applications.
Publication date 08/22/2019
Share this material on social networks and leave your opinion in the comments below.
