Module 1 ppt BIG DATA ANALYTICS_NOTES FOR MCA

BIG DATA

1. Introduction to Big Data In order to understand 'Big Data', we first need to know what ' data ' is. Oxford dictionary defines 'data' as – "The quantities, characters, or symbols on which operations are performed by a computer, which may be stored and transmitted in the form of electrical signals and recorded on magnetic, optical, or mechanical recording media. “

2. Introduction to Big Data 'Big Data' is also a data but with a huge size . 'Big Data' is a term used to describe collection of data that is huge in size and yet growing exponentially with time. In short, s uch a data is so enormous and complex that none of the traditional data management tools are able to store it or process it efficiently .

3. Different types of sources of data for Big Data analytics Structured Data : This type of data is organized in a fixed format, such as data stored in relational databases or spreadsheets. Unstructured Data : This type of data does not have a predetermined format, such as text documents, images, videos, and social media posts. Semi-Structured Data : This type of data is a combination of structured and unstructured : data, such as XML or JSON files. Sensor Data This type of data comes from various sensors and devices, such as GPS trackers, IoT devices, and machine sensors. Social Media Data : This type of data comes from various social media platforms like Facebook, Twitter, and Instagram. Web Data : This type of data comes from web pages, web logs, and web APIs. Machine Data : This type of data comes from machines and systems, such as log files, system performance data, and telemetry data. Mobile Data : This type of data comes from mobile devices, such as app usage data, location data, and mobile sensor data. Public Data : This type of data comes from various public sources, such as government data, weather data, and census data. Transactional Data : This type of data comes from various transactional systems, such as sales data, financial transactions, and customer interactions.

4. Applications of Big Data Analytics Marketing and customer targeting: Big data analytics can be used to analyze customer behavior and preferences, allowing companies to better target their marketing efforts and personalize their messaging to individual customers. fraud detection and prevention : Big data analytics can be used to detect and prevent fraudulent activities in industries such as finance, insurance, and healthcare, by analyzing patterns and anomalies in data to identify potential fraudulent behavior. Healthcare analy t ics : Big data analytics can be used in the healthcare industry to analyze patient data, optimize treatment plans, and improve patient outcomes. It can also be used to predict and prevent diseases, reduce healthcare costs, and improve overall quality of care.

5. Applications of Big data Analytics Predictive maintenance : Big data analytics can be used to analyze equipment data and predict when machines are likely to fail, allowing companies to perform maintenance before a breakdown occurs and minimize downtime. Social media analytics : Big data analytics can be used to analyze social media data, track customer sentiment, and identify trends and opportunities for engagement. This can help companies improve their social media strategy and enhance their online presence. Risk management : Big data analytics can be used to analyze data to identify potential risks and opportunities, allowing companies to make more informed decisions and mitigate potential threats. Retail analytics : Big data analytics can be used in the retail industry to analyze customer data, optimize pricing strategies, and improve inventory management. It can also be used to identify trends and patterns in consumer behavior to drive sales and enhance customer experience.

6. Characteristics Of Big Data Volume : Big data refers to datasets that are massive in size, typically ranging from terabytes to petabytes in scale. Velocity : Big data is generated and processed at a high speed, with data being created, collected, and analyzed in real-time. Variety : Big data consists of diverse types of data, including structured, semi-structured, and unstructured data, such as text, images, videos, and sensor data. Veracit y: Big data can be noisy or incomplete, requiring advanced data cleaning and processing techniques to ensure accuracy and reliability. Value : Big data has the potential to provide valuable insights and opportunities for organizations, enabling them to make informed decisions and gain a competitive edge. Complexity : Big data is complex in nature, often requiring sophisticated technologies and techniques, such as machine learning and artificial intelligence, to extract meaningful insights from the data. Scalabilit y: Big data systems are designed to be scalable, allowing organizations to easily store, process, and analyze large volumes of data as their needs grow. Flexibilit y: Big data systems are flexible, allowing organizations to easily integrate and analyze data from various sources, formats, and structures. Privacy and security : Big data raises concerns around privacy and security, as organizations must ensure that sensitive data is protected and comply with regulations such as GDPR. Real-time analysis : Big data systems are capable of performing real-time analysis, enabling organizations to quickly respond to changing market conditions and make data-driven decisions.

7. Analytics Process Model

8. ANALYTICS PROCESS MODEL Problem Definition. Identification of data source. Selection of Data. Data Cleaning. Transformation of data . Analytics. Interpretation and Evaluation.

9. ANALYTICS PROCESS MODEL Problem Definition. Problem identification and definition: The problem is a situation that is judged as something that needs to be corrected. It is the job of the analyst to make sure that the right problem Is solved . Problem can be identified through : Comparative / Benchmarking studies. Benchmarkin g i s comparin g one’s business processe s and performanc e metric s t o industr y bests and best practices from other companies. Performance reporting. Assessment of present performance against goals objectives. SWOT analysis.

10. Analytics Process Model Depending on type of the problem , source data need to be identified. As data is the key ingredient to any analytical exercise and the selection of data will have a deterministic impact on the analytical models that we are building Few Data Collection Technique: Using data that has already been collected by others. Systematically selecting and watching characteristics of people, object and events. Oral questioning of respondents either individually or a group. Facilitating free discussions on specific topics with selected group of participants.

11. Analytics Process Model Data Cleaning : Before a formal data analysis can be conducted, the analyst must know how many cases are there in the data set, what variables are included, how many missing observations are there and what general hypothesis the data is likely to suffer. Analyst commonly use visualization for data exploration because it allows users to quickly and simply view most of the relevant features of their data set.

12. Analytics Process Model Data transformation : it is the process of converting data from one format, such as a database file, XML document or Excel spreadsheet, into another. Transformations typically involve converting a raw data source into a cleansed, validated and ready-to-use format.

13. Analytics Process Model Big Data Analytics comes in many different types, each serving a different purpose: Descriptive Analytics:  This type helps us understand past events. In social media, it shows performance metrics, like the number of likes on a post. Diagnostic Analytics:  In Diagnostic analytics delves deeper to uncover the reasons behind past events. In healthcare, it identifies the causes of high patient re-admissions. Predictive Analytics:  Predictive analytics forecasts future events based on past data. Weather forecasting, for example, predicts tomorrow’s weather by analyzing historical patterns. Prescriptive Analytics:  This type not only predicts outcomes but also suggests actions to optimize them. In e-commerce, it might recommend the best price for a product to maximize profits. Real-time Analytics:  Real-time analytics processes data instantly. In stock trading, it helps traders make quick decisions based on current market conditions. Spatial Analytics:  Spatial analytics focuses on location data. For city planning, it optimizes traffic flow using data from sensors and cameras to reduce congestion. Text Analytics:  Text analytics extracts insights from unstructured text data. In the hotel industry, it can analyze guest reviews to improve services and guest satisfaction.

14. Analytics Process Model Evaluation and Interpretation of Big Data Analytics Data quality assessment : Before diving into analysis, it is important to assess the quality of the data. This involves identifying any inconsistencies, errors, or missing values in the data that could affect the accuracy of the analysis. Data preprocessing : Data preprocessing involves cleaning, transforming, and organizing the data to make it suitable for analysis. This step helps in ensuring that the data is in a format that can be easily interpreted and analyzed. Choosing the right analytical tools : There are various analytical tools available for big data analytics, such as machine learning algorithms, data mining techniques, and visualization tools. It is important to choose the right tools that are best suited for the specific requirements of the analysis. Data visualization : Data visualization is a powerful tool for interpreting big data as it helps in identifying patterns, trends, and outliers in the data. Visualization techniques such as charts, graphs, and heat maps can provide a clear and easy-to-understand representation of the data. Interpretation of results: Once the analysis is complete, it is important to interpret the results in a meaningful way. This involves deriving actionable insights from the data, identifying key trends and patterns, and making recommendations based on the findings. Validation and verif ication: It is important to validate and verify the results of the analysis to ensure their accuracy and reliability. This can be done through cross-validation, sensitivity analysis, and comparison with external data sources.

15. Analytical Model Requirements in big data analytics Scalability : The analytical model should be able to handle large volumes of data efficiently and effectively. It should be able to scale up or down based on the size of the dataset being analyzed. Performance: The model should be able to deliver high performance in terms of processing speed and accuracy. It should be able to provide insights in a timely manner to support decision making. Accuracy: The model should be able to provide accurate results and predictions based on the data being analyzed. It should be able to minimize errors and provide reliable information for decision making. Flexibility: The model should be able to adapt to changing data patterns and trends. It should be able to adjust its algorithms and parameters to accommodate new data and changes in the data environment. Interpretability : The model should be able to provide clear and understandable insights and explanations to users. It should be able to explain how it arrived at certain results and predictions in a way that is easily understandable.

16. Analytical Model Requirements in big data analytics Integration: The model should be able to integrate with other data sources and systems to provide a comprehensive view of the data being analyzed. It should be able to connect with different data sources, tools, and platforms to access relevant data for analysis. Security: The model should have robust security features to protect the data being analyzed. It should ensure the confidentiality, integrity, and availability of the data being processed and provide secure access to authorized users. Automation : The model should have automated features to streamline the analysis process and reduce the manual effort required. It should be able to automate data preprocessing, feature selection, model building, and evaluation to accelerate the analysis process. Model explainability : The model should be able to explain how it arrived at its conclusions and predictions in a transparent and interpretable way. This helps build trust in the model and its results. Support for real-time analytics : The model should have the capability to perform real-time analysis and provide insights on streaming data. It should be able to handle data in motion and deliver results in near real-time to support quick decision making.

17. Sampling in Big Data Sampling in big data analytics is the process of selecting a representative subset of data from a larger dataset for analysis. This is done to reduce the computational burden and speed up the analysis process, especially when dealing with massive datasets that are too large to analyze in their entirety. There are various sampling techniques that can be used in big data analytics, including random sampling, stratified sampling, cluster sampling, and systematic sampling. Each technique has its own advantages and disadvantages, and the choice of sampling method depends on the specific goals of the analysis. Sampling in big data analytics is important because it allows analysts to draw meaningful conclusions from a smaller subset of data, without having to analyze the entire dataset. It can help in identifying patterns, trends, and anomalies in the data, and can provide insights that can be used for decision-making and strategic planning. However, it is important to ensure that the sampling process is done carefully and in a way that accurately represents the overall dataset to avoid bias and misleading results.

18. Sampling Types Simple random sampling.  Software is used to randomly select subjects from the whole population. Stratified sampling.  Analysts create subsets within the population based on a common factor and then collect random samples from each subgroup. Cluster sampling.  Analysts divide the population into subsets (clusters), based on a defined factor. Then, they analyze a random sampling of clusters. Multistage sampling.  This approach is a more complicated form of cluster sampling. It divides the larger population into multiple clusters and then breaks those clusters into second-stage clusters, based on a secondary factor. The secondary clusters are then sampled and analyzed. This staging could continue as multiple subsets are identified, clustered and analyzed. Systematic sampling.  A sample is created by setting an interval at which to extract data from the larger population. For example, an analyst might select every 10th row in a spreadsheet of 2,000 items to create a sample size of 200 rows to analyze.

19. Data elements required for Big Data Analytics Structured data: This type of data can be easily organized and stored in a traditional relational database format. It includes data such as numbers, dates, and text. Unstructured data : This type of data does not have a pre-defined structure and can come in a variety of formats, such as images, videos, social media posts, and emails. Semi-structured data : This type of data falls somewhere in between structured and unstructured data. It may have a partial structure (such as tags or labels) that can be used to organize and analyze the data. Time-series data : This type of data includes information that is collected over time, such as stock prices, weather data, and sensor readings. Geospatial data : This type of data includes information about physical locations and can be used to analyze trends and patterns related to geographic locations. Text data : This type of data includes written text, such as social media posts, customer reviews, and emails. Text data can be analyzed using natural language processing techniques. Multimedia data : This type of data includes images, videos, and audio files. Multimedia data can be analyzed using computer vision and audio processing techniques. Sensor data: This type of data includes information collected from sensors, such as temperature sensors, GPS devices, and accelerometers.

20. Data Exploration Data exploration is a crucial step in big data analytics that involves examining and analyzing data to discover patterns, trends, and insights. The goal of data exploration is to gain a deeper understanding of the data and extract valuable information that can be used to make informed decisions. There are several techniques and tools that can be used for data exploration in big data analytics, including: Descriptive statistics: Descriptive statistics such as mean, median, mode, and standard deviation can provide an overview of the data and help identify outliers and anomalies. Data visualization: Data visualization techniques such as charts, graphs, and heatmaps can help in identifying patterns and trends in the data that may not be easily apparent from raw data. Data mining: Data mining techniques such as clustering, classification, and anomaly detection can be used to uncover hidden patterns and relationships in the data.

21. Data Exploration Dimensionality reduction: Dimensionality reduction techniques such as principal component analysis (PCA) and t-distributed stochastic neighbor embedding (t-SNE) can help in reducing the complexity of the data and visualizing high-dimensional data in a lower-dimensional space. Text analysis: Text analysis techniques such as sentiment analysis, topic modeling, and natural language processing (NLP) can be used to extract insights from unstructured text data. Overall, data exploration plays a key role in big data analytics as it helps in uncovering valuable insights and making informed decisions based on the data.

22. Exploratory Statistical Analysis Exploratory Statistical Analysis (ESA) is a critical first step in analyzing any dataset. It involves investigating the characteristics and patterns present in the data before making any formal statistical inferences or modeling assumptions. ESA aims to understand the structure of the data, identify trends, detect outliers, and formulate hypotheses for further investigation. Here are some common techniques used in exploratory statistical analysis: Descriptive Statistics : Calculate summary statistics such as mean, median, mode, variance, standard deviation, skewness, and kurtosis to summarize the central tendency, dispersion, and shape of the data distribution. Data Visualization : Create visual representations of the data using histograms, box plots, scatter plots, heatmaps, and other graphical techniques to identify patterns, trends, and outliers. Correlation Analysis : Examine the relationships between pairs of variables using correlation coefficients (e.g., Pearson correlation, Spearman rank correlation) to identify potential associations.

23. Exploratory Statistical Analysis Dimensionality Reduction : Apply techniques such as principal component analysis (PCA) or t-distributed stochastic neighbor embedding (t-SNE) to reduce the dimensionality of the data while preserving its essential structure for visualization and analysis. Clustering Analysis : Use clustering algorithms such as k-means clustering or hierarchical clustering to identify groups or clusters within the data based on similarity measures. Outlier Detection : Identify data points that deviate significantly from the rest of the data distribution using statistical methods (e.g., z-scores) or anomaly detection algorithms. Distributional Analysis : Assess the distributional properties of variables using probability plots, goodness-of-fit tests (e.g., Kolmogorov-Smirnov test), and normality tests (e.g., Shapiro-Wilk test). Time Series Analysis : Explore temporal patterns and trends in time series data using techniques such as autocorrelation analysis, decomposition, and seasonality detection. Data Imputation : Handle missing values in the dataset using techniques such as mean imputation, median imputation, or sophisticated imputation methods like k-nearest neighbors (KNN) or multiple imputation. Feature Engineering : Transform or create new features from existing variables to improve the performance of predictive models or enhance the interpretability of the data.

24. Missing Values Handling missing values in big data analysis requires careful consideration due to the large volume and complexity of the data. Here are some strategies commonly employed for dealing with missing values in big data analysis: Identify Missing Values : Begin by identifying and understanding the extent of missing values in the dataset. This may involve examining summary statistics or using visualization techniques to visualize missing value patterns. Deletion : One approach is to simply delete rows or columns with missing values. While this can be straightforward, it may lead to loss of valuable information, especially in big data where every data point counts.

25. Missing Values Imputation : Imputation involves replacing missing values with estimated values based on the observed data. Common imputation techniques include: Mean/Median Imputation : Replace missing values with the mean or median of the non-missing values in the same column. This method is simple but may not capture the true underlying distribution. Mode Imputation : Replace missing categorical values with the mode (most frequent value) of the respective column. Regression Imputation : Use regression models to predict missing values based on other variables in the dataset. K-Nearest Neighbors (KNN) Imputation : Estimate missing values based on the values of the nearest neighbors in the feature space. Multiple Imputation : Generate multiple imputed datasets to capture uncertainty in the imputation process, particularly useful in predictive modeling.

26. Outlier Detection and Treatment Outlier detection and treatment are crucial steps in big data analytics to ensure the reliability and accuracy of analytical results. Here's a comprehensive approach to handling outliers in big data: Identify Outliers : Start by identifying outliers in the dataset. Outliers are data points that deviate significantly from the rest of the data and may distort statistical analyses and machine learning models. Common techniques for outlier detection include: Univariate Methods: Use statistical measures such as z-scores, interquartile range (IQR), or Tukey's fences to identify outliers based on individual variables. Multivariate Methods: Apply techniques like principal component analysis (PCA) or Mahalanobis distance to detect outliers in multivariate data. Density-Based Methods: Utilize density-based clustering algorithms like DBSCAN (Density-Based Spatial Clustering of Applications with Noise) to identify outliers as data points in low-density regions. .

27. Outlier Detection and Treatment Visualize Outliers : Visualize the distribution of data and potential outliers using histograms, box plots, scatter plots, or heatmaps. Visualization helps in understanding the nature and extent of outliers in the data. Handle Outliers : Remove Outliers : In some cases, it may be appropriate to remove outliers from the dataset, especially if they are due to data entry errors or measurement errors. However, this approach should be used judiciously as it may lead to loss of valuable information Transform Data : Apply transformations such as logarithmic transformation or Box-Cox transformation to reduce the impact of outliers and make the data more normally distributed. Winsorization : it involves capping or clipping extreme values by replacing them with the nearest non-outlier value. This approach mitigates the effect of outliers without completely removing them .

28. Outlier Detection and Treatment Imputation : Impute outlier values with more plausible estimates based on statistical methods or domain knowledge. For example, use robust statistical measures like median or trimmed mean for imputation instead of the mean. Model-Based Approaches : Use robust statistical models or machine learning algorithms that are less sensitive to outliers, such as robust regression or support vector machines (SVMs) with robust kernels. Ensemble Methods : Combine multiple outlier detection techniques or models to improve detection accuracy and robustness. Automate Outlier Detection : Implement automated outlier detection pipelines using distributed computing frameworks like Apache Spark or scalable machine learning libraries. This allows for efficient processing of large-scale datasets and real-time detection of outliers in streaming data.

29. Outlier Detection and Treatment Monitor Data Quality : Establish data quality monitoring processes to continuously track outliers and assess their impact on analytical results. Regularly evaluate the effectiveness of outlier detection and treatment methods and refine them as needed.

30. Standardizing Data Labels Standardizing data labels in big data analytics is crucial for ensuring consistency, interoperability, and accuracy in data analysis and decision-making. Here's a systematic approach to standardizing data labels : Define a Data Label Standardization Strategy : Establish a clear strategy and guidelines for standardizing data labels. Consider factors such as industry standards, regulatory requirements, organizational conventions, and best practices in data management. Develop a Data Dictionary : Create a comprehensive data dictionary that documents the meaning, format, and usage of each data label or variable in the dataset. The data dictionary serves as a centralized reference for data standardization efforts and promotes transparency and collaboration among data stakeholders.

31. Standardizing Data Labels Identify Common Data Labels : Identify common data labels or variables across different datasets and data sources within the organization. Standardize these labels to ensure consistency and alignment in data analysis and reporting processes. Establish Naming Conventions : Develop naming conventions for data labels that follow a consistent format and structure. Use descriptive, intuitive names that convey the meaning and context of the data, making it easier for users to interpret and analyze the data. Map and Transform Data Labels : Map data labels from diverse sources to a standardized set of labels using mapping tables or transformation rules. Convert variations in label names, abbreviations, or synonyms to the standardized format to ensure uniformity and comparability across datasets.

32. Standardizing Data Labels Implement Data Labeling Tools : Utilize data labeling tools or software platforms to automate the process of standardizing data labels. These tools can facilitate bulk renaming, mapping, and transformation of data labels, streamlining the standardization workflow. Validate and Review Standardized Labels : Validate the standardized data labels through peer review, data profiling, and quality assurance checks. Ensure that the standardized labels accurately represent the underlying data and adhere to the established standards and conventions. Document Standardization Processes : Document the standardization processes, methodologies, and decisions made during the data labeling process. Maintain documentation to facilitate knowledge sharing, onboarding of new team members, and auditing of data standardization practices.

33. Standardizing Data Labels Update and Maintain Standards : Regularly review and update the data label standardization standards to adapt to evolving business requirements, data sources, and analytical needs. Continuously refine and improve standardization processes based on feedback and lessons learned. Train Users and Stakeholders : Provide training and guidance to users and stakeholders on the standardized data labels and how to interpret and use them effectively in data analysis and decision-making processes.

34. Categorization In big data analytics, categorization plays a vital role in organizing and structuring large volumes of data to extract meaningful insights and drive decision-making. Here's how categorization is utilized in the context of big data analytics : Data Preprocessing : Before analysis can begin, raw data often needs to be preprocessed. Categorization techniques are used to clean, normalize, and standardize data, ensuring consistency and compatibility across different sources. Feature Engineering : Categorization helps in feature engineering, where raw data is transformed into meaningful features that can be used by machine learning algorithms for predictive modeling. This involves categorizing data attributes, identifying relevant patterns, and creating new features based on these categories.

35. Categorization Segmentation : Categorization is used to segment data into distinct groups or clusters based on similarities or shared characteristics. This segmentation enables analysts to focus on specific subsets of data for deeper analysis and targeted decision-making. Classification and Prediction : Categorization techniques such as classification algorithms are employed to assign data instances to predefined categories or predict the category of new data instances based on past observations. This is particularly useful in tasks such as customer segmentation, fraud detection, and sentiment analysis. Taxonomies and Ontologies : In some cases, hierarchical categorization is used to create taxonomies or ontologies that organize data into nested categories or concepts. These structures provide a framework for organizing and navigating complex data sets, facilitating better understanding and exploration.

36. Categorization Topic Modeling : Categorization methods like topic modeling are used to identify themes or topics within unstructured text data. By automatically categorizing documents or text passages into topics, analysts can uncover underlying patterns, trends, and insights. Anomaly Detection : Categorization techniques are applied in anomaly detection to distinguish between normal and abnormal behavior within data sets. By categorizing data points as either typical or anomalous, analysts can identify potential outliers or deviations that may indicate fraud, errors, or unusual patterns.

37. Module-2 What is Hadoop Hadoop is a framework that uses distributed storage and parallel processing to store and manage big data. It is the software most used by data analysts to handle big data. Overview of Hadoop Doug Cutting and Mike Cafarella were the creator of HADOOP. Doug and Mike Cafarella were building a project called “Nutch” with the goal of creating a large Web index. They saw the MapReduce and GFS papers from Google, which were obviously super relevant to the problem Nutch was trying to solve. They integrated the concepts from MapReduce and GFS into Nutch; then later these two components were pulled out to form the genesis of the Hadoop project.

38. Characteristics of Hadoop Open Source Hadoop is an open-source project, and its code is available can be modified according to its business requirements. Distributed Processing As data is stored in a distributed manner in HDFS across the cluster and data is processed in parallel on a cluster of nodes. Faster Hadoop is extremely good at high volume batch processing because of its ability to do parallel processing. Fault Tolerance The data sent to one individual node and the same data also replicates on other nodes in the same cluster. If the individual node fails to process the data the other nodes in the same cluster will be available to process the data.

39. Characteristics of Hadoop Scalability Hadoop is highly storage platform as it can store and distribute very large data sets across the hundreds of systems/servers that operate in parallel. Flexibility Hadoop manages whether structured or unstructured, encoded or formatted or any other type of data. Business can use Hadoop to derive the valuable business insights from data sources such as social media, email conversations. Easy to Use No need of client to deal with distributed computing, the framework takes care of all the things. Hadoop is easy to use.

40. Hadoop Architecture

41. Hadoop Architecture MapReduce MapReduce nothing but just like an Algorithm or a  data structure  that is based on the YARN framework. The major feature of MapReduce is to perform the distributed processing in parallel in a Hadoop cluster which Makes Hadoop working so fast. When you are dealing with Big Data, serial processing is no more of any use. HDFS HDFS(Hadoop Distributed File System) is utilized for storage. It is mainly designed for distributed file system design. HDFS is designed in such a way that it believes more in storing the data in a large chunk of blocks rather than storing small data blocks.  Data in HDFS is always stored in terms of blocks. So the single block of data is divided into multiple blocks of size 128MB which is default and you can also change it manually. 

42. Hadoop Architecture YARN(Yet Another Resource Negotiator) YARN is a Framework on which MapReduce works. YARN performs 2 operations that are Job scheduling and Resource Management. The Purpose of Job schedular is to divide a big task into small jobs so that each job can be assigned to various slaves in a Hadoop cluster and Processing can be Maximized. Job Scheduler also keeps track of which job is important, which job has more priority, dependencies between the jobs and all the other information like job timing, etc. And the use of Resource Manager is to manage all the resources that are made available for running a Hadoop cluster.  Hadoop common or Common Utilities Hadoop common or Common utilities are nothing but our java library and java files or we can say the java scripts that we need for all the other components present in a Hadoop cluster. These utilities are used by HDFS, YARN, and MapReduce for running the cluster. Hadoop Common verify that Hardware failure in a Hadoop cluster is common so it needs to be solved automatically in software by Hadoop Framework.

43. Data Discovery Data discovery in Hadoop refers to the process of exploring and analyzing large volumes of data stored in a Hadoop ecosystem, typically using tools like Apache Hive, Apache Pig, Apache Spark, or Apache Drill. Here's an overview of how data discovery typically works in Hadoop: Data Ingestion : Data is ingested into the Hadoop cluster from various sources such as relational databases, log files, sensor data, social media feeds, etc. This data is often stored in the Hadoop Distributed File System (HDFS) or Hadoop-compatible storage systems like Amazon S3. Data Cataloging and Metadata Management : To facilitate data discovery, metadata management tools are used to catalog the ingested data. These tools extract metadata such as data types, file formats, schema information, and data lineage, and store them in a central repository.

44. Data Discovery Querying and Analysis : Analysts and data scientists can then use query engines and analytics tools like Hive, Pig, Spark SQL, or Drill to explore the data. These tools provide SQL-like interfaces or programming APIs for querying and analyzing the data stored in Hadoop. Visualization and Reporting : Once the data is queried and analyzed, the results can be visualized using tools like Tableau, Power BI, or Apache Superset. Visualization helps in understanding trends, patterns, and insights hidden in the data. Machine Learning and Advanced Analytics : Advanced analytics techniques such as machine learning, predictive analytics, and natural language processing can be applied to the data to derive deeper insights and make predictions. Data Governance and Security : Data discovery processes must adhere to data governance policies to ensure data security, compliance, and privacy. Access controls, encryption, and auditing mechanisms are implemented to protect sensitive data.

45. Open Source Technology for Big Data Analytics Open-source technologies have revolutionized the field of big data analytics, offering cost-effective and flexible solutions for processing, storing, and analyzing vast amounts of data. Here are some popular open-source tools commonly used in big data analytics: Apache Hadoop : Hadoop is perhaps the most well-known open-source framework for distributed storage and processing of large datasets. It consists of the Hadoop Distributed File System (HDFS) for storage and MapReduce for processing. Apache Spark : Spark is a fast and general-purpose cluster computing system that provides in-memory data processing capabilities. It's known for its speed and ease of use, offering APIs for various programming languages like Scala, Java, Python, and R. Apache Kafka : Kafka is a distributed streaming platform that is often used for building real-time data pipelines and streaming applications. It's highly scalable, fault-tolerant, and provides strong durability guarantees. Apache Flink : Flink is a powerful stream processing framework with capabilities for batch processing as well. It offers low-latency processing, exactly-once

46. Open Source Technology for Big Data Analytics Apache Storm : Storm is another stream processing framework designed for real-time analytics. It's highly scalable and fault-tolerant, suitable for processing large streams of data with low latency. Elasticsearch : It is a distributed search and analytics engine commonly used for full-text search, log analytics, and other use cases requiring fast search capabilities over large datasets. Apache Cassandra : Cassandra is a distributed NoSQL database known for its scalability and high availability. It's designed to handle large amounts of data across multiple nodes without a single point of failure. Apache HBase : It is a distributed, scalable, and consistent NoSQL database built on top of Hadoop's HDFS. It's often used for random, real-time read/write access to big data . These are just a few examples of the many open-source technologies available for big data analytics. Each tool has its strengths and weaknesses, so the choice often depends on specific use cases, requirements, and preferences.

47. Cloud and Big data Cloud computing and big data often go hand in hand, as the scalability and flexibility of cloud platforms are well-suited for storing, processing, and analyzing large volumes of data. Here's how cloud technologies intersect with big data: Scalability : Cloud platforms such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) offer virtually unlimited scalability. This means you can easily scale up or down your computing resources based on the volume of data you need to process or analyze. Storage : Cloud providers offer various storage solutions tailored for big data, such as Amazon S3, Azure Blob Storage, and Google Cloud Storage. These services provide highly durable, scalable, and cost-effective storage for large datasets. Compute : Cloud platforms offer a wide range of computing services suitable for big data analytics, including virtual machines (VMs), containers (e.g., Docker), and serverless computing (e.g., AWS Lambda, Azure Functions). This allows you to deploy and run your analytics workloads efficiently without worrying about managing underlying infrastructure.

48. Cloud and Big data Managed Services : Cloud providers offer managed big data services that abstract the complexity of setting up and managing infrastructure. For example, AWS provides Amazon EMR (Elastic MapReduce) for running Hadoop, Spark, and other big data frameworks, while Azure offers Azure HDInsight and Google Cloud provides Dataproc . Analytics Services : Cloud platforms offer various analytics services for processing and analyzing big data, such as AWS Athena for interactive query analysis, Azure Synapse Analytics for data warehousing and analytics, and Google BigQuery for serverless data warehousing and SQL analytics.

49. Cloud and Big data Machine Learning and AI : Cloud providers offer machine learning and AI services that leverage big data for training and inference, such as AWS SageMaker , Azure Machine Learning, and Google AI Platform. These services allow you to build and deploy machine learning models at scale using your big data. Integration : Cloud platforms provide integration capabilities with popular big data tools and frameworks, allowing you to seamlessly connect your existing infrastructure with cloud services. For example, you can easily integrate Apache Kafka with cloud services like AWS Kinesis and Azure Event Hubs for real-time data streaming.

50. Predictive Analytics Predictive analytics in big data involves using statistical techniques, machine learning, and data mining to analyze historical and current data to make predictions about future events. Here are key aspects of how predictive analytics is applied within the real world of big data : 1. Data Collection and Preparation Volume : Big data involves massive amounts of structured and unstructured data collected from various sources such as social media, sensors, transaction records, and more. Variety : Data comes in different formats including text, images, videos, and more. Velocity : The speed at which data is generated and processed is critical. Veracity : Ensuring data quality and accuracy is essential for reliable predictions.

51. Predictive Analytics 2. Techniques and Algorithms Regression Analysis : Used for predicting continuous outcomes based on historical data. Classification Algorithms : Such as decision trees, random forests, and support vector machines, used for categorizing data into predefined classes. Clustering : Groups data points into clusters with similar characteristics, useful in market segmentation and customer profiling. Time Series Analysis : Analyzes data points collected or recorded at specific time intervals to forecast future values.

52. Predictive Analytics 3. Machine Learning and AI Supervised Learning : Algorithms are trained on labeled data, making predictions based on learned patterns. Unsupervised Learning : Algorithms identify hidden patterns or intrinsic structures in input data without labeled responses . 4. Applications in Various Domains Marketing and Sales : Predict customer behavior, optimize marketing campaigns, and personalize offers. Finance : Fraud detection, credit scoring, and investment predictions. Healthcare : Predict disease outbreaks, patient outcomes, and optimize treatment plans. Manufacturing : Predictive maintenance, quality control, and optimizing production processes.

53. Mobile Business Intelligence and Big Data Definition Mobile BI : Refers to the ability to access BI-related data such as dashboards, reports, and analytics on mobile devices like smartphones and tablets. It enhances decision-making by providing real-time access to data. Key Features Real-Time Access : Mobile BI applications provide up-to-date information, crucial for timely decision-making. User-Friendly Interfaces : Designed for touchscreens, these interfaces are intuitive and easy to navigate. Interactive Dashboards : Allow users to drill down into data, customize views, and interact with visualizations.

54. Mobile Business Intelligence and Big Data Integration with Big Data Big Data Characteristics Volume : The sheer amount of data generated from various sources. Velocity : The speed at which data is created and processed. Variety : Different types of data (structured, unstructured, semi-structured). Veracity : The accuracy and trustworthiness of data.

55. Mobile Business Intelligence and Big Data Synergy between Mobile BI and Big Data Enhanced Data Access : Mobile BI applications can tap into large datasets from big data environments, providing users with comprehensive insights. Advanced Analytics : Combining mobile BI with big data allows for advanced analytics like predictive modeling and machine learning, accessible on mobile devices. Scalability : Big data technologies ensure that mobile BI platforms can scale to handle increasing data volumes without performance degradation.

56. Mobile Business Intelligence and Big Data Applications and Benefits Sales and Marketing : Sales teams can access customer data, sales performance metrics, and market trends in real-time, enhancing customer interactions and sales strategies. Finance : Real-time financial dashboards enable tracking of key financial indicators, budgeting, and forecasting from anywhere. Healthcare : Medical professionals can access patient records, diagnostic data, and treatment plans on the move, improving patient care.

57. Crowdsourcing Analytics Definition Crowdsourcing Analytics : The process of obtaining analytical insights and solutions by engaging a crowd of people, often through online platforms, to contribute data analysis, predictive modeling, and problem-solving efforts. Mechanisms Open Calls : Publicly inviting individuals or groups to participate in analytics challenges or projects. Competitions : Hosting contests where participants submit their analytical solutions for evaluation and rewards. Collaborative Platforms : Using online platforms where users can collaborate, share data, and build on each other’s work.

58. Crowdsourcing Analytics Cost-Effectiveness Reduced Costs : Compared to hiring full-time experts or consultants, crowdsourcing can be a more cost-effective way to tackle complex analytics problems. Scalability : Easily scales to handle large datasets and complex problems by distributing the workload across many participants. Speed and Efficiency Faster Results : Multiple people working simultaneously can analyze data and generate insights more quickly than a single team. 24/7 Progress : Contributors from different time zones ensure continuous progress on analytics projects.

59. Crowdsourcing Analytics Applications Business Intelligence Market Research : Gathering insights from a wide audience to understand market trends and consumer behavior. Product Development : Collecting feedback and ideas from potential users to improve products and services. Healthcare Medical Research : Crowdsourcing data analysis for medical research, such as genomic studies or disease pattern identification. Public Health : Analyzing data from various sources to track disease outbreaks and health trends. Finance Risk Management : Engaging experts to analyze financial data for risk assessment and mitigation strategies. Investment Strategies : Crowdsourcing analytics to develop and refine investment models and trading algorithms.

60. Inter and Trans Firewall Analytics Inter-Firewall Analytics Definition : Analyzing traffic that flows between different firewalls within a network infrastructure. This includes traffic between different organizational units, data centers, or between corporate networks and external partners. Trans-Firewall Analytics Definition : Analyzing traffic that passes through a single firewall. This includes monitoring inbound and outbound traffic to detect anomalies, threats, and inefficiencies.

61. Inter and Trans Firewall Analytics Techniques and Tools Traffic Monitoring and Analysis Packet Inspection : Deep Packet Inspection (DPI) analyzes the data part (and possibly also the header) of a packet as it passes an inspection point. Flow Analysis : NetFlow , sFlow , and IPFIX provide data on traffic flows, helping identify patterns and anomalies . Firewall Management Tools Policy Management : Tools like AlgoSec , FireMon , and Tufin help manage and optimize firewall policies across multiple devices, ensuring consistency and reducing the risk of misconfigurations. Change Management : Tracks and audits changes to firewall configurations to ensure they comply with security policies and regulatory requirements.