Evolution & Introduction to Big data-2.pptx

skgfykd5is75fykuflkydoydoydlydlhflyyflhgghlcmhdkgdhlfykfhlfoydlhdlhclhhfhlfhlxlhfhlfhlfhlfhlghlglhglhchlcluchlclhclhcjlclhcjlgjlghlgluflufhlfhlclhfglAspect extraction is a crucial step in **aspect-based sentiment analysis (ABSA)**, where the goal is to identify and analyze sentiments toward specific aspects of a product or service (e.g., "battery life" in a review of a smartphone). Two common approaches for aspect extraction are **frequency-based methods** and **syntactic dependency-based methods**.

### 1. Frequency-Based Aspect Extraction

#### Methodology:
- **Word Frequency Analysis:** In this approach, the algorithm identifies aspects by analyzing word frequencies in a corpus of texts. Words or phrases that appear frequently in a dataset are considered potential aspects (e.g., "camera" or "battery" in smartphone reviews).
- **Co-occurrence Analysis:** Words that frequently appear in conjunction with sentiment words (like “good” or “bad”) are also considered as potential aspects. These pairs are often seen as aspect-sentiment pairs.
- **Term Frequency-Inverse Document Frequency (TF-IDF):** It ranks words by their frequency but down-weights terms that are common across all documents, focusing on the more unique and relevant aspects for each review.

#### Advantages:
- **Simplicity:** Frequency-based methods are simple to implement, requiring only basic text preprocessing and statistical analysis.
- **Scalability:** Since this approach relies on counting words, it scales well to large datasets.
- **Domain Independence:** These methods do not require a deep understanding of language structure, making them adaptable to different domains without significant customization.

#### Limitations:
- **Context Insensitivity:** These methods do not account for the context in which words are used. For example, the word "battery" might not always refer to the aspect "battery life" in a smartphone review but also to other contexts like "battery backup."
- **Inability to Handle Rare Aspects:** Aspects that occur infrequently may be ignored or under-represented in the extraction process, leading to loss of important information.
- **No Sentiment-Aspect Association:** Frequency-based methods typically do not capture the relationship between an aspect and its corresponding sentiment unless additional analysis is done.

#### Scenarios Where It Works Well:
- **Large Datasets with Repetitive Aspect Mentions:** In domains like product reviews where aspects are frequently mentioned (e.g., "screen" or "battery" in electronics reviews), frequency-based methods are effective.
- **Exploratory Analysis:** When you need a quick, high-level view of the most common aspects without delving into complex linguistic structures.

### 2. Syntactic Dependency-Based Aspect Extraction

#### Methodology:
- **Dependency Parsing:** This method uses syntactic parsers to analyze the grammatical structure of sentences and identify dependencies between words. For example, in a sentence like

1. Evolution & Introduction to Big data

2. data Data is like a collection of facts or information. The quantity of data created by humans is quickly increasing every year as a result of the introduction of new technology, gadgets, and communication channels such as social networking sites.

3. Big data Big Data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, sharing, transfer, analysis, and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to “spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime and determine real time roadway traffic conditions.”

4. Big data examples Every day, 500+ terabytes of fresh data are absorbed into the Facebook systems. This information is mostly gathered through photo and video uploads, message exchanges, and the posting of comments, among other things. In 30 minutes of flying time, a single Jet engine may create 10+ gigabytes of data. With thousands of flights every day, the amount of data generated can amount to several Petabytes. Every day, the Fresh York Stock Exchange creates around a terabyte of new trading data.

5. The size chart of data

6. What’s making so much data? Sources: People, machine, organization. More people carrying data-generating devices (Mobile phones with Facebook, GPS, Cameras, etc.) Organization-Generated data is often referred to as Transactional data.

7. Big data analytics Big data analytics is like using a giant magnifying glass to look at a huge amount of information to find patterns, trends, or important details. It's a way to make sense of a lot of data and learn something useful from it. Suppose you have a favorite online store where you buy toys, clothes, and games. The store collects data on what everyone buys, what they look at, and even what they search for but don't buy. This data is very large because so many people are shopping every day—this is "big data.“ Big data analytics is when the store uses powerful computers and smart programs to look at all this information. They try to find patterns, like: Which toys are most popular during the holidays? What types of clothes do kids in different parts of the country like best? What games are often bought together?

8. Big data analytics example Netflix Recommendations: When you watch shows and movies on Netflix, it keeps track of what you watch, when you watch it, and what you like or dislike. Netflix uses big data analytics to look at all this information from millions of viewers. It finds patterns and uses them to recommend new shows and movies that you might like based on what you've watched before. This makes your viewing experience more enjoyable and helps you discover new content.

9. Big data characteristics

10. Big data characteristics In the year 2001, MetaGroup (now known as Gartner) which is an analytics firm introduced data scientists and analysts to the 3Vs of Data. The 3 Vs are Volume, Velocity, and Variety. But over a period of time, Data analytics saw a change in how data was captured and processed. The observation includes that data was growing so rapidly in size, later this data came to be known as Big data. 5V’s of Big data 1. Volume 2. Value 3. Velocity 4. Variety 5. Validity / Veracity

11. Big data characteristics 1. Volume Big data volume can be defined as the amount of data that is produced. The volume of data produced is also dependent on the size of the data. In today’s technological world data is generated from various sources in different formats. Data formats are in the form of the word, excel documents, PDFs, media content such as images, videos, etc. are produced at a great pace. It is becoming challenging for enterprises to store and process data using the conventional methods of business intelligence and analytics. Enterprises need to implement modern business intelligence tools to effectively capture, store and process such huge amounts of data in real life. Some interesting facts: Today, around 2.7 Zettabytes of data exist in the digital world. Walmart handles more than 1 million customer transactions every hour, which is imported into databases estimated to contain approximately 2.5 petabytes of data.

12. Big data characteristics 2. Value Value refers to the benefits that big data can provide, and it relates directly to what organizations can do with that collected data. Being able to pull value from big data is a requirement, as the value of big data increases significantly depending on the insights that can be gained from it. Organizations can use big data tools to gather and analyze the data, but how they derive value from that data should be unique to them. Tools like Apache  Hadoop  can help organizations store, clean and rapidly process this massive amount of data. A great example of big data value can be found in the gathering of individual customer data. When a company can profile its customers, it can personalize their experience in  marketing and sales , improving the efficiency of contacts and garnering greater customer satisfaction.

13. Big data characteristics 3. Velocity Velocity means “How fast produce Data”. Now-a-days, Organizations or Human Beings or Systems are generating huge amounts of Data at very fast rate. An organization that uses big data will have a large and continuous flow of data that's being created and sent to its end destination. Data could flow from sources such as machines, networks, smartphones or social media. Velocity applies to the speed at which this information arrives -- for example, how many social media posts per day are ingested -- as well as the speed at which it needs to be digested and analyzed -- often quickly and sometimes in near real time. As an example, in healthcare, many medical devices today are designed to monitor patients and collect data. From in-hospital medical equipment to wearable devices, collected data needs to be sent to its destination and analyzed quickly. In some cases, however, it might be better to have a limited set of collected data than to collect more data than an organization can handle -- because this can lead to slower data velocities.

14. Big data characteristics 4. Variety Variety means “Different forms of Data”. Now-a-days, Organizations or Human Beings or Systems are generating very huge amount of data at very fast rate in different formats. We will discuss in details about different formats of Data soon. The volume and velocity of data add value to an organization or business, but the diverse data types collected from varied data sources are also an important factor of Big data. Big data is generally classified as structured, semi-structured, or unstructured data. 1. Structured data Structured data is one whose format, length, and volume are clearly defined. 2. Semi-structured data Semi-structured data is one that may partially conform to a specific data format. 3. Unstructured data Unstructured data is unorganized data that doesn’t conform with the traditional data formats. Data generated via digital and social media such as images, videos, etc , is unstructured data.

15. Big data characteristics 5. Validity / Veracity The Validity and Veracity of Big data can be described as the assurance of quality or credibility of the collected data. Since Big data is vast and involves so many data sources, it is the possibility that not all the collected data is accurate and of good quality. Hence, when processing   big data sets, it is important to check the validity of the data before proceeding with further analysis. Questions like Can you trust the data that you have collected? Is the data reliable enough? , etc. need to be entertained. Hence, before processing the data for further analysis, it is important to check the validity of the data.

16. The Challenges For most organizations, Big Data analysis is a challenge. Consider the sheer volume of data and the different formats of the data(both structured and unstructured data) that is collected across the entire organization and the many different ways different types of data can be combined, contrasted and analyzed to find patterns and other useful business information. The first challenge is in breaking down data silos to access all data an organization stores in different places and often in different systems. A second challenge is in creating platforms that can pull in unstructured data as easily as structured data. This massive volume of data is typically so large that it's difficult to process using traditional database and software methods.

17. Benefits of Big Data Processing Ability to process Big Data brings in multiple benefits, such as- 1. Businesses can utilize outside intelligence while taking decisions. 2. Access to social data from search engines and sites like Facebook, twitter are enabling organizations to fine tune their business strategies. 3. Improved customer service (Traditional customer feedback systems are getting replaced by new systems designed with Big Data technologies. 4. Improved customer service (In these new systems, Big Data and natural language processing technologies are being used to read and evaluate consumer responses. 5. Early identification of risk to the product/services, if any 6. Better operational efficiency

18. Why is Big Data Important? • Cost Savings Big data helps in providing business intelligence that can reduce costs and improve the efficiency of operations. Processes like quality assurance and testing can involve many complications particularly in industries like biopharmaceuticals and nanotechnologies • Time Reductions Companies may collect data from a variety of sources using real-time in-memory analytics. Tools like Hadoop enable businesses to evaluate data quickly, allowing them to make swift decisions based on their findings. • Understand the market conditions Businesses can benefit from big data analysis by gaining a better grasp of market conditions. Analyzing client purchase behavior, for example, enables businesses to discover the most popular items and develop them appropriately. This allows businesses to stay ahead of the competition.

19. Why is Big Data Important? • Social Media Listening’s Companies can perform sentiment analysis using Big Data tools. These enable them to get feedback about their company, that is, who is saying what about the company. Companies can use Big data tools to improve their online presence • Using Big Data Analytics to Boost Customer Acquisition and Retention . Customers are a crucial asset that each company relies on. Without a strong consumer base, no company can be successful. However, even with a strong consumer base, businesses cannot ignore market rivalry. It will be difficult for businesses to succeed if they do not understand what their consumers desire. It will be difficult for businesses to succeed if they do not understand what their consumers desire. It will result in a loss of customers, which will have a negative impact on business growth. Businesses may use big data analytics to detect customer-related trends and patterns. Customer behavior analysis is the key to a successful business.

20. Why is Big Data Important? • Using Big Data Analytics to Solve Advertisers Problem and Offer Marketing Insights All company activities are shaped by big data analytics. It allows businesses to meet client expectations. Big data analytics aids in the modification of a company's product range. It guarantees that marketing initiatives are effective. • Big Data Analytics as a Driver of Innovations and Product Development Companies may use big data to innovate and revamp their goods.

21. Applications of BIG DATA

22. Applications of BIG DATA Here is the list of top Big Data applications in today’s world: • Big Data in Retail • Big Data in Healthcare • Big Data in Education • Big Data in E-commerce • Big Data in Media and Entertainment • Big Data in Finance • Big Data in Travel Industry • Big Data in Telecom • Big Data in Automobile

23. Applications of BIG DATA 1. Big Data in Retail The retail industry is the one that faces the most fierce competition of all. Retailers constantly hunt for ways that will give them a competitive edge over others. Customers are the real king sounds legit for the retail industry in particular. For retailers to thrive in this competitive world, they need to understand their customers in a better way. If they are aware of their customers’ needs and how to fulfill those needs in the best possible way, then they know everything. Check how Big Data act as a weapon for retailers to connect with their customers – Big Data in Retail: Through advanced analysis of their customer’s data, retailers are now able to understand them from every angle possible. They gather this data from various sources such as social media, loyalty programs, etc.

24. Applications of BIG DATA Even a minute detail about any customer has now become significant for them. They are now closer to their customers than they have ever been. This empowers them to provide customers with more personalized services and predict their demands in advance. This helps them in building a loyal customer base. Some of the biggest names in the retail world like Walmart, Sears and Holdings, Costco, Walgreens, and many more now have Big Data as an integral part of their organizations. A study by the National Retail Federation estimated that sales in November and December are responsible for as much as 30% of retail annual sales.

25. Applications of BIG DATA 2. Big Data in Healthcare Big Data and healthcare are an ideal match. It complements the healthcare industry better than anything ever will. The amount of data the healthcare industry has to deal with is unimaginable. Gone are the days when healthcare practitioners were incapable of harnessing this data. From finding a cure to cancer to detecting Ebola and much more, Big Data has got it all under its belt and researchers have seen some life-saving outcomes through it. Big Data and analytics have given them the license to build more personalized medications. Data analysts are harnessing this data to develop more and more effective treatments. Identifying unusual patterns of certain medicines to discover ways for developing more economical solutions is a common practice these days. Smart wearables have gradually gained popularity and are the latest trend among people of all age groups. This generates massive amounts of real-time data in the form of alerts which helps in saving the lives of the people.

26. Applications of BIG DATA 3. Big Data in Education When you ask people about the use of the data that an educational institute gathers, the majority of the people will have the same answer that the institute or the student might need it for future references. Even you had the same perception about this data, didn’t you? But the fact is, this data holds enormous importance. Big Data is the key to shaping the future of the people and has the power to transform the education system for better. Some of the top universities are using Big Data as a tool to renovate their academic curriculum. Additionally, universities can even track the dropout rates of the students and are taking the required measures to reduce this rate as much as possible.

27. Applications of BIG DATA 4. Big Data in E-commerce One of the greatest revolutions this generation has seen is that of E-commerce. It is now part and parcel of our routine life. Whenever we need to buy something, the first thought that provokes our mind is E-commerce. And not your surprise, Big Data has been the face of it. Some of the biggest E-commerce companies of the world like Amazon, Flipkart, Alibaba, and many more are now bound to Big Data and analytics is itself an evidence of the level of popularity Big Data has gained in recent times. Big Data is now as important as anyone else in these organizations. Amazon, the biggest Ecommerce firm in the world and one of the pioneers of Big Data and analytics, has Big Data as the backbone of its system. Flipkart, the biggest E-commerce firm in India, has one of the most robust data platforms in the country. Big Data’s recommendation engine is one of the most amazing applications the Big Data world has ever witnessed. It furnishes the companies with a 360-degree view of its customers. Companies then suggest customers accordingly. Customers now experience more personalized services than they have ever had. Big Data has completely redefined people’s online shopping experiences

28. Applications of BIG DATA 5. Big Data in Media and Entertainment Media and Entertainment industry is all about art and employing Big Data in it is a sheer piece of art. Art and science are often considered to be the two completely contrasting domains but when employed together, they do make a deadly duo and Big Data’s endeavors in the media industry are a perfect example of it. Viewers these days need content according to their choices only. Content that is relatively new to what they saw the previous time. Earlier the companies broadcasted the Ads randomly without any kind of analysis. But after the advent of Big Data analytics in the industry, companies now are aware of the kind of Ads that attracts a customer and the most appropriate time to broadcast it for seeking maximum attention. Customers are now the real heroes of the Media and entertainment industry - courtesy to Big Data and Analytics

29. Applications of BIG DATA Example Spotify Spotify has nearly 96 million users and all these users generate tremendous amount of data. User behaviour includes songs played, repeatedly used playlists, likes, shares and search history, all of which represents the big data with respect to Spotify. Spotify analyses this big data for suggesting songs to its users. All of you might have come across the recommendation list which is made available to each one of you by Spotify, each one of you will have a totally different recommendation list based on your likes, your past history or the songs you like listening to and your playlist. Recommendations systems are data filtering tools. They collect data and then filter them using algorithms. This system has the ability to accurately predict what a user would like to listen to next with the help of big data analytics.

30. Applications of BIG DATA 6. Big Data in Finance The functioning of any financial organization depends heavily on its data and to safeguard that data is one of the toughest challenges any financial firm faces. Data has been the second most important commodity for them after money. Even before Big Data gained popularity, the finance industry was already conquering the technical field. In addition to it, financial firms were among the earliest adopters of Big Data and Analytics. Digital banking and payments are two of the most trending buzzwords around and Big data has been at the heart of it. Big Data is bossing the key areas of financial firms such as fraud detection, risk analysis, algorithmic trading, and customer contentment. This has brought much-needed fluency in their systems. They are now empowered to focus more on providing better services to their customers rather than focusing on security issues. Big Data has now enhanced the financial system with answers to its hardest of the challenges.

31. Applications of BIG DATA 7. Big Data in Travel Industry While Big Data is spreading like wildfire and various industries have been cooking its food with it, the travel industry was a bit late to realize its worth. Better late than never though. Having a stress-free traveling experience is still like a daydream for many. And now Big Data’s arrival is like a ray of hope, that will mark the departure of all the hindrances in our smooth traveling experience. Through Big Data and analytics, travel companies are now able to offer more customized traveling experience. They are now able to understand their customer’s requirements in a much-enhanced way. From providing them with the best offers to be able to make suggestions in real-time, Big Data is certainly a perfect guide for any traveler. Big Data is gradually taking the window seat in the travel industry.

32. Applications of BIG DATA 8. Big Data in Telecom The telecom industry is the soul of every digital revolution that takes place around the world. With the ever-increasing popularity of smartphones, it has flooded the telecom industry with massive amounts of data. And this data is like a goldmine, telecom companies just need to know how to dig it properly. Through Big Data and analytics, companies are able to provide the customers with smooth connectivity, thus eradicating all the network barriers that the customers have to deal with. Companies now with the help of Big Data and analytics can track the areas with the lowest as well as the highest network traffics and thus doing the needful to ensure hassle-free network connectivity. Big Data alike other industries have helped the telecom industry to understand its customers pretty well. Telecom industries now provide customers with offers as customized as possible.

33. Applications of BIG DATA 9. Big Data in Automobile “A business like an automobile, has to be driven, in order to get results.” B.C. Forbes And Big Data has now taken complete control of the automobile industry and is driving it smoothly. Big Data is driving the automobile industry towards some unbelievable and never before results. The automobile industry is on a roll and Big Data is its wheels or I must say Big Data has given wings to it. Big Data has helped the automobile industry achieve things that were beyond our imaginations. From analyzing the trends to understanding the supply chain management, from taking care of its customers to turning our wildest dream of connected cars a reality, Big Data is well and truly driving the automobile industry crazy.

34. Other use case of big data Analytics Big data analytics is used for risk management : BDO (Banco de Oro), a Philippine banking company uses big data analytics for risk management. Risk management is an important aspect for any organization especially in the field of banking. Risk management comprises a series of measures which are employ to implement any sort of unauthorized activities. BDO adopted big data analytics ; Identifying fraudulent activities and discrepancies is easier using big data analytics. Thus the organization was able to narrow down the list of suspects using big data analytics.

35. Other use case of big data Analytics Big data analytics is used for product development and innovations: Rolls-Royce manufactures massive jet engines. These engines are used by airlines and armed forces across the world. They used big data analytics for developing and innovating this engine, new product is always developed by trial and error method. The company uses big data analytics to analyze how good the engine design is and if there has to be any more improvement (based on the previous model and future demands). Big data analytics is used here in designing a product of higher quality. Using big data analytics the company can save a lot of time.

36. Other use case of big data Analytics Big data analytics helps in quicker and better decision making in organizations: Starbucks uses big data analytics for important decisions. They decide the location of their new outlet using big data analytics. Choosing a right is an important factor for any organization. The wrong location will not be able to attract the required amount of customers. Various factors are involved in choosing the right location for a new outlet such as population demographics, accessibility of the location, competition in the vicinity, economic factors, parking adequacy and so on. The business grows if the right location is chosen wisely by considering the above parameters.

37. Other use case of big data Analytics

38. Life cycle of big data analytics

39. Life cycle of big data analytics Stage 1 : Business case evaluation: The Big Data analytics lifecycle begins with a business case, which defines the reason and goal behind the analysis. Identify the motive behind the analysis. Understand the problem or opportunity to be addressed. Define the goals and objectives of the analysis. We need to understand why they are analysing. So, that we know how to do it and what are the different parameters that have to be looked into. Example: A retail company wants to analyze customer purchase behavior to increase sales and improve customer satisfaction.

40. Life cycle of big data analytics Stage 2: Identification of data: After which we will look into the various data sources from where we can gather all the data which will be required for analysis. Here, a broad variety of data sources are identified. Once we get the required data we will have to see the data that we receive is fit for analysis or not. Not all the data that we receive will have meaningful information , some of them will surely be corrupt data. To remove corrupt data we will pass this data to filtering stage. Identify various data sources. Gather data from different sources (e.g. customer surveys, transactional data, social media, sensors). Evaluate the quality and relevance of the data. Example: The retail company identifies the following data sources: + Customer surveys + Transactional data from point-of-sale systems + Social media data from Twitter and Facebook + Sensor data from in-store cameras and RFID tags.

41. Life cycle of big data analytics Stage 3: Data filtering: At the filtering stage all the corrupt data will be removed. All of the identified data from the previous stage is filtered here to remove corrupt data. Remove corrupt, incomplete, or irrelevant data. Clean and preprocess the data to make it analysis-ready. Example: The retail company removes duplicate records, handles missing values, and corrects formatting errors in the transactional data.

42. Life cycle of big data analytics Stage 4: Data extraction: Data that is not compatible with the tool is extracted and then transformed into a compatible form. Before making data to fit for analysis, We still have figure out which data will be compatible with the tool that we will be using for analysis, if we find data which is incompatible we will first extracted and then transform it to a compatible form depending on the tool that we use. Extract relevant data from various sources. Transform data into a compatible format for analysis. Handle data incompatibilities and inconsistencies. Example: The retail company extracts customer demographic data from surveys and transforms it into a format compatible with their analytics tool.

43. Life cycle of big data analytics Stage 5: Data aggregation: In this stage, data with the same fields across different datasets are integrated. Integrate data from different sources. Combine data with similar fields or attributes. Create a unified view of the data. Example: The retail company aggregates customer purchase data from different stores and online platforms to create a single, unified view of customer behavior.

44. Life cycle of big data analytics Stage 6: Data analysis: It is very important stage in the life cycle of big data analytics, here in this step d ata is evaluated using analytical and statistical tools to discover useful information. Apply analytics techniques to extract insights from the data. Use statistical models, machine learning algorithms, or data mining techniques. Identify patterns, trends, and correlations in the data. Example: The retail company uses clustering analysis to identify customer segments based on purchase behavior and demographics.

45. Life cycle of big data analytics Stage 7: Visualization of data: The result of the data analysis stage is graphically communicated using various tools. With tools  like  Tableau ,  Power BI , and QlikView , Big Data analysts can produce graphic visualizations of the analysis. Present insights in a clear and actionable way. Use data visualization tools to create reports, dashboards, and charts. Communicate findings to stakeholders. Example: The retail company creates an interactive dashboard to visualize customer purchase trends and preferences.

46. Life cycle of big data analytics Stage 8: Final analysis result: This is the last step of the Big Data analytics lifecycle, where the final results of the analysis are made available to business stakeholders who will take action. Analysis results will then be made available to different stakeholders for various decision making. Interpret the results and generate actionable insights. Identify opportunities for improvement or innovation. Develop recommendations for business stakeholders. Example: The retail company identifies an opportunity to increase sales by 10% by targeting a specific customer segment with personalized promotions.

47. Types of big data analytics

48. 1. Descriptive Analytics This summarizes past data into a form that people can easily read. This helps in creating reports, like a company’s revenue, profit, sales, and so on. Use Case: The Dow Chemical Company analyzed its past data to increase facility utilization across its office and lab space. Using  descriptive analytics,  Dow was able to identify underutilized space. This space consolidation helped the company save nearly US $4 million annually.

49. 2. Diagnostic Analytics This is done to understand what caused a problem in the first place. Techniques like drill-down,  data mining , and data recovery are all examples. Organizations use diagnostic analytics because they provide an in-depth insight into a particular problem. Use Case: An e-commerce company’s report shows that their sales have gone down, although customers are adding products to their carts. This can be due to various reasons like the form didn’t load correctly, the shipping fee is too high, or there are not enough payment options available. This is where you can use diagnostic analytics to find the reason. 

50. 3. Predictive Analytics This type of analytics looks into the historical and present data to make predictions of the future. Predictive analytics uses  data mining , AI, and  machine learning  to analyze current data and make predictions about the future. It works on predicting customer trends, market trends, and so on. Use Case: PayPal determines what kind of precautions they have to take to protect their clients against fraudulent transactions. Using  predictive analytics,  the company uses all the historical payment data and user behavior data and builds an algorithm that predicts fraudulent activities.

51. 4. Prescriptive Analytics This type of analytics prescribes the solution to a particular problem. Perspective analytics works with both descriptive and predictive analytics. Most of the time, it relies on  AI and machine learning. Use Case: Prescriptive analytics can be used to maximize an airline’s profit. This type of analytics is used to build an algorithm that will automatically adjust the flight fares based on numerous factors, including customer demand, weather, destination, holiday seasons, and oil prices.

52. Best Practices for Big Data Analytics Define Clear Objectives Choose the Right Tools and Technology Ensure Data Quality Optimize Data Storage and Processing Implement Robust Security Measures Embrace Data Visualization and Interpretation Enable Collaboration and Cross-Functional Insights Continuous Monitoring and Iteration Scale Responsibly Invest in Skills and Training

53. Best Practices for Big Data Analytics In today's data-driven world, organizations are increasingly leveraging big data analytics to gain valuable insights and drive informed decision-making. However, successfully managing and deriving meaningful insights from large volumes of data requires adherence to several best practices. Here are key strategies and considerations for optimizing big data analytics projects: 1. Define Clear Objectives Before diving into big data analytics, it's crucial to establish clear objectives and define what insights you aim to derive. Whether it's improving operational efficiency, understanding customer behavior, or optimizing marketing strategies, a well-defined goal will guide your analytics efforts.

54. Best Practices for Big Data Analytics 2. Choose the Right Tools and Technologies: Selecting the appropriate tools and technologies is essential for the success of big data projects. Consider factors such as scalability, performance, integration capabilities, and ease of use when choosing platforms like Hadoop, Spark, or cloud-based solutions such as AWS or Azure. 3. Ensure Data Quality: High-quality data is fundamental to accurate analytics. Implement data quality checks and cleansing processes to identify and rectify inconsistencies, errors, and duplicates. Data governance practices should be in place to maintain data integrity throughout its lifecycle.

55. Best Practices for Big Data Analytics 4. Optimize Data Storage and Processing: Efficient data storage and processing are critical for timely analytics. Leverage distributed computing frameworks like Hadoop or Spark to handle large datasets effectively. Utilize data partitioning, indexing, and compression techniques to optimize storage and retrieval. 5. Implement Robust Security Measures: Protecting data privacy and ensuring security are paramount in big data analytics. Employ encryption, access controls, and authentication mechanisms to safeguard sensitive information. Adhere to regulatory compliance requirements such as GDPR or CCPA.

56. Best Practices for Big Data Analytics 6. Embrace Data Visualization and Interpretation: Transform complex data into actionable insights through visualization techniques like charts, graphs, and dashboards. User-friendly visualization tools enable stakeholders to understand and interpret data more effectively. 7. Enable Collaboration and Cross-Functional Insights: Encourage collaboration between data scientists, analysts, and business stakeholders. Foster a culture of data-driven decision-making by ensuring insights are shared across departments and used to drive strategic initiatives. 8. Continuous Monitoring and Iteration: Big data analytics is an iterative process. Continuously monitor performance metrics, validate models, and refine algorithms based on new data and changing business needs. Embrace agile methodologies to adapt quickly to evolving requirements.

57. Best Practices for Big Data Analytics 9. Scale Responsibly : Plan for scalability from the outset to accommodate growing data volumes and analytical demands. Leverage cloud-based solutions for elasticity and on-demand resources, ensuring cost efficiency while maintaining performance. 10. Invest in Skills and Training: Develop a skilled workforce capable of harnessing the power of big data analytics. Provide ongoing training and upskilling opportunities to keep pace with emerging technologies and industry trends. By following these best practices, organizations can unlock the full potential of big data analytics, driving innovation, optimizing operations, and gaining a competitive edge in today's data-centric landscape.

58. Validating–The Promotion of the Value of Big Data "Validating the Promotion of the Value of Big Data" refers to the process of confirming and demonstrating the benefits and effectiveness of big data initiatives within an organization or broader context. This involves several key steps: 1. Defining Objectives and Metrics : Clearly outlining what the big data initiative aims to achieve and how success will be measured. 2. Collecting and Analyzing Data : Gathering relevant data and using analytical techniques to process and interpret this data to generate insights. 3. Demonstrating Results : Presenting findings that show how big data has contributed to improved decision-making, operational efficiencies, cost savings, or other valuable outcomes. 4. Comparative Analysis : Comparing results before and after the implementation of big data solutions to highlight the added value. 5. Feedback and Improvement : Gathering feedback from stakeholders to refine and enhance big data strategies, ensuring continuous improvement. 6. Communicating Value : Effectively communicating the benefits and value derived from big data initiatives to stakeholders, including management, employees, and possibly customers. In essence, it’s about proving that investing in big data technologies and methodologies has a tangible positive impact and is worth the resources and efforts expended.

59. Validating–The Promotion of the Value of Big Data 1. Case Studies Examples of Organizations Using Big Data Successfully: - Amazon: Uses big data for personalized recommendations, inventory management, and dynamic pricing, resulting in increased sales and customer satisfaction. - Netflix: Analysis viewer preferences and behaviour to provide personalized content recommendations, leading to higher viewer retention and engagement. - Walmart: Leverages big data to optimize supply chain management and predict product demand, improving efficiency and reducing costs. - Google: Utilizes big data for improving search algorithms and targeted advertising, enhancing user experience and increasing ad revenue. - Uber: Uses data analytics to optimize routing, reduce wait times, and improve driver and passenger experience.

60. Validating–The Promotion of the Value of Big Data 2. Metrics and KPIs: Key Performance Indicators to Measure Big Data Impact: Key Performance Indicators (KPIs)Definition: Key Performance Indicators (KPIs) are measurable values that demonstrate how effectively an organization is achieving key business objectives. They help organizations track progress, identify areas for improvement, and make data-driven decisions. - Revenue Growth: Measure the increase in sales or revenue directly attributed to big data initiatives. - Cost Savings: Track reductions in operational costs due to improved efficiency and decision-making. - Customer Satisfaction: Use customer feedback, Net Promoter Score (NPS), or customer retention rates to gauge improvements in customer experience. - Operational Efficiency: Monitor key metrics like time to market, supply chain efficiency, and resource utilization. - Innovation: Assess the number of new products or services developed as a result of big data insights. - Accuracy and Speed: Evaluate the accuracy of predictions and the speed of data processing and decision-making.

61. Validating–The Promotion of the Value of Big Data 3. Cost-Benefit Analysis: Comparing Costs and Benefits of Big Data Solutions: - Costs: - Initial investment in technology and infrastructure (e.g., hardware, software, cloud services). - Ongoing operational costs (e.g., data storage, processing, maintenance). - Hiring and training costs for skilled personnel (e.g., data scientists, analysts). - Compliance and security costs to protect data and adhere to regulations.

62. Validating–The Promotion of the Value of Big Data 3 . Cost-Benefit Analysis: Comparing Costs and Benefits of Big Data Solutions: - Benefits: - Increased revenue from personalized marketing, better product recommendations, and new business opportunities. - Cost savings from optimized operations, reduced waste, and improved efficiency. - Enhanced customer loyalty and satisfaction through personalized experiences and better service. - Competitive advantage gained from insights and innovations driven by data analytics. - Improved risk management and decision-making capabilities. Example: - Investment in a Big Data Analytics Platform: - Cost: $1 million (technology, infrastructure, personnel). - Benefit: $3 million increase in revenue from personalized marketing and $500,000 in cost savings from optimized supply chain management and Net Benefit: $2.5 million.

63. Validating–The Promotion of the Value of Big Data 4. Feedback and Iteration: Gathering Feedback and Iterating on Strategies: - Stakeholder Feedback: Regularly collect input from stakeholders (e.g., employees, customers, partners) to understand their needs and experiences. - Continuous Improvement: Use feedback to refine and enhance big data strategies, ensuring they remain aligned with business goals. - Agile Methodologies: Adopt agile practices to quickly adapt to changing requirements and incorporate new insights. - Performance Monitoring: Continuously track and evaluate key metrics and KPIs to identify areas for improvement. - Pilot Projects: Run small-scale pilot projects to test new ideas and approaches before full-scale implementation. - Regular Reviews: Schedule regular reviews of big data initiatives to assess progress, address challenges, and celebrate successes.

64. THE PROMOTION OF THE VALUE OF BIG DATA O ne way to do this is to review the difference between what is being said about big data and what is being done with bigdata. A scan of existing content on the ―value of big data sheds interesting light on what is being promoted as the expected result of big data analytics and, more interestingly, how familiar those expectations sound. A good example is provided within an economic study on the value of big data(titled ―Data Equity—Unlocking the Value of Big Data), undertaken and published by the Center for Economics and Business Research (CEBR) that speaks to the cumulative value of: • optimized consumer spending as a result of improved targeted customer marketing; • improvements to research and analytics within the manufacturing sectors to lead to new product development; • improvements in strategizing and business planning leading to innovation and new start-up companies; • predictive analytics for improving supply chain management to optimize stock management, replenishment, and forecasting; improving the scope and accuracy of fraud detection.

65. THE PROMOTION OF THE VALUE OF BIG DATA Curiously, these are exactly the same types of benefits promoted by business intelligence and D atawarehouse tools vendors and system integrators for the past 15_20 years, namely: • Better targeted customer marketing • Improved product analytics • Improved business planning • Improved supply chain management • Improved analysis for fraud, waste, and abuse Further articles, papers, and vendor messaging on big data reinforce these presumptions.

66. Perception and Quantification of Value Drilldown into the question of value and whether using big data significantly contributes to adding value to the organization by: • Increasing revenues: As an example, an expectation of using a recommendation engine would be to increase same-customer sales by adding more items into the market basket. • Lowering costs: As an example, using a big data platform built on commodity hardware for ETL would reduce or eliminate the need for more specialized servers used for data staging, thereby reducing the storage footprint and reducing operating costs. • Increasing productivity: Increasing the speed for the pattern analysis and matching done for fraud analysis helps to identify more instances of suspicious behaviour faster, allowing for actions to be taken more quickly and transform the organization from being focused on recovery of funds to proactive prevention of fraud • Reducing risk: Using a big data platform or collecting many thousands of streams of automated sensor data can provide full visibility into the current state of a power grid, in which unusual events could be rapidly investigated to determine if a risk of an imminent outage can be reduced.

67. Traditional data v/s Big data

68. Understanding Big Data Storage Traditional approaches to storing and processing big data often involve a combination of relational databases and batch processing systems. Here's a breakdown of these traditional methods: Relational Databases: SQL Databases Batch Processing Systems: Data Warehouses ETL (Extract, Transform, Load) Processes

69. Traditional approach of storing and processing big data Relational Databases: SQL Databases: Relational databases like MySQL, PostgreSQL, and Oracle use Structured Query Language (SQL) to manage and query data. They store data in tables with predefined schemas, where each table has rows and columns. Usage : These databases are great for structured data and ensure data integrity and consistency through ACID (Atomicity, Consistency, Isolation, Durability) properties. For example: A bank might use a relational database to manage customer accounts, transactions, and personal information, ensuring accurate and reliable financial records.

70. Traditional approach of storing and processing big data Batch Processing Systems: Data Warehouses : Data warehouses like Amazon Redshift, Google BigQuery , and Teradata are specialized databases optimized for analytical queries and reporting. They aggregate data from various sources into a central repository. Usage : These systems are used for large-scale data analysis, allowing organizations to run complex queries and generate reports. For example: A retail company might use a data warehouse to analyze sales data from different stores and generate monthly performance reports.

71. Traditional approach of storing and processing big data ETL (Extract, Transform, Load) Processes : In a traditional approach usually the data that is being generated out of the organizations, the financial institutions such as banks or stock markets and the hospitals is given as the input to ETL Systems. An ETL System would then extract its data and transform this data that is it would convert this data into a proper format and finally load this data onto the database; now the end users can generate reports and performs analytics by acquiring this data. But as the data grows it becomes a very challenging task to manage and process this data using the traditional approach. This is the one of the fundamental drawback of using the traditional approach.

72. Drawbacks of using traditional approach Scalability Issues : Traditional databases and data warehouses often struggle to scale horizontally. As data volumes grow, it becomes challenging to manage and process data efficiently. For Example : A growing e-commerce platform may find its relational database struggling to handle increasing transaction volumes and user data. Performance Bottlenecks : Batch processing systems can lead to significant delays in data availability. Data is processed in large chunks at scheduled intervals, which isn't suitable for real-time analytics. For Example : A financial institution may experience delays in generating end-of-day reports due to the time required for batch processing. Data Integration Challenges: Complex ETL Processes : Extracting, transforming, and loading data from various sources into a traditional data warehouse can be complex, time-consuming, and error-prone. For Example : A global retail chain may find it challenging to integrate sales data from different regions and formats, leading to inconsistencies and inaccuracies in the data warehouse.

73. Drawbacks of using traditional approach Inefficient Handling of Big Data: Volume, Variety, Velocity : Traditional approaches aren't well-suited to handle the three V's of big data (volume, variety, and velocity). They struggle with the massive scale, diverse data types, and rapid data generation rates of modern data environments. Example : An Internet of Things (IoT) company collecting data from millions of sensors in real-time may find traditional databases inadequate for storing and processing such high-velocity and high-volume data. Lack of Real-Time Processing: Delayed Insights : Traditional batch processing systems do not support real-time data processing, leading to delays in generating insights and taking action. Example : A logistics company tracking real-time locations of shipments may not be able to provide timely updates and optimize routes if relying on traditional batch processing.

74. Modern Solutions to Overcome These Drawbacks To overcome these limitations, modern big data technologies like Hadoop, Spark, and NoSQL databases have emerged. These tools offer: Scalability : They can handle vast amounts of data by distributing storage and processing across multiple nodes. Real-Time Processing : Technologies like Spark and Kafka enable real-time data analysis and streaming. Flexibility : NoSQL databases like MongoDB and Cassandra can handle unstructured and semi-structured data, providing more flexibility in data management. By leveraging these modern technologies, organizations can efficiently store, process, and analyze big data, driving better insights and decision-making.

75. Characteristics of Big Data Applications The characteristics of Big Data applications refer to the fundamental attributes and features that define how these applications handle, process, and analyze large and complex datasets. These characteristics help distinguish Big Data applications from traditional data applications. Understanding these characteristics is crucial for designing, implementing, and managing effective Big Data solutions. Here is a more detailed explanation of each characteristic: 1. Volume : Refers to the vast amounts of data generated every second. Examples : Social media posts, sensor data from IoT devices, transaction records. Requires significant storage capacity and efficient data management techniques. Implications : Requires storage solutions that can handle large datasets, such as distributed databases and cloud storage.

76. Characteristics of Big Data Applications 2. Velocity : The speed at which data is generated, collected, and processed. Examples : Stock market data, real-time analytics for web applications, streaming services. Necessitates real-time or near real-time data processing capabilities to derive timely insights and make quick decisions. Implications : Technologies like Apache Kafka and Apache Spark are often used. 3. Variety : The different types of data from various sources. Examples : Structured data (databases), semi-structured data (XML, JSON), unstructured data (text, images, videos). Requires flexible data storage, integration, and analysis techniques to handle different data formats and sources. Implications : Requires flexible data integration, storage solutions, and sophisticated analytics techniques to handle diverse data types.

77. Characteristics of Big Data Applications 4. Veracity : The uncertainty and trustworthiness of data. Examples : Incomplete data, noisy data, biases in data. Data cleaning, validation, and error correction are essential to ensure accurate and reliable analysis. Implications : Data cleaning and preprocessing are crucial. Techniques like data validation, normalization, and error correction are used to improve data quality. 5. Value : The potential insights and benefits derived from data. Examples : Business intelligence, customer behavior analysis, predictive maintenance. Emphasizes the need for advanced analytics to transform raw data into valuable information. Implications : Focus on extracting actionable insights and achieving ROI from Big Data initiatives. Involves data mining, machine learning, and advanced analytics.

78. Characteristics of Big Data Applications 6. Complexity : The difficulty of managing and analyzing large and diverse datasets. Examples : Integrating data from multiple sources, dealing with different data formats. Requires sophisticated tools, technologies, and methodologies to handle the complexity effectively. Implications : Requires sophisticated tools and technologies, such as ETL (Extract, Transform, Load) processes, data lakes, and complex event processing systems. 7. Scalability : The ability to scale up or down in response to data volume and processing demands. Examples : Expanding storage capacity, increasing computational power. Ensures that applications can handle increasing data loads efficiently without performance degradation. Implications : Utilizes scalable architectures like cloud computing and distributed systems to handle growing data needs efficiently.

79. Characteristics of Big Data Applications 8. Distributed Computing : Processing data across multiple machines or nodes. Examples : Hadoop, Spark. Enables efficient data processing by distributing workloads across a cluster of computers. Implications : Enables handling large datasets by distributing the processing workload. Requires knowledge of distributed algorithms and fault tolerance. 9. Real-Time Processing : The capability to process and analyze data as it is generated. Examples : Fraud detection systems, real-time recommendation engines. Critical for applications that require quick decision-making, such as fraud detection, online recommendations, and monitoring systems. Implications : Requires low-latency data processing frameworks and real-time analytics tools.

80. Characteristics of Big Data Applications 10. Advanced Analytics : Use of complex algorithms and techniques to analyze data. Examples : Machine learning, artificial intelligence, predictive modeling. Enables deep data analysis to uncover hidden trends and make accurate predictions. Implications : Involves specialized skills and tools to implement and interpret advanced analytical models. 11. Data Governance : Policies and processes to manage data security, privacy, and compliance. Examples : GDPR compliance, data encryption, access control. Protects sensitive information and ensures that data management practices comply with legal and regulatory requirements. Implications : Ensures that data is managed properly to protect sensitive information and adhere to legal regulations. Involves establishing data governance frameworks and protocols.

81. Big Data Analytics Tools Hadoop  - helps in storing and analyzing data. MongoDB - used on datasets that change frequently Talend - used for data integration and management Cassandra - a distributed database used to handle chunks of data Spark - used for real-time processing and analyzing large amounts of data STORM - an open-source real-time computational system Kafka - a distributed streaming platform that is used for fault-tolerant storage