DATA SCRAPING AND WEB Scrapping.....pptx

It is an important for creating data scraping his own most important thing that we use in today's world because the data scraping place in important role for the return and it is because if there is a no data we cannot be able to do anything any stuff in this today is world that's why the data scrapping has been playing a very important role if the important rules admin not completed by the data scrapping than the data scrapping is just 10 news list there are two many steps skipping if the slapping is not done then the skepping is not use thisown most important thing that we use in today's world because the data scraping place in important role for the return and it is because if there is a no data we cannot be able to do anything any stuff in this today is world that's why the data scrapping has been playing a very important role if the important rules admin not completed by the data scrapping than the data scrapping is just 10 news list there are two many steps skipping if the slapping is not done then the skepping i
own most important thing that we use in today's world because the data scraping place in important role for the return and it is because if there is a no data we cannot be able to do anything any stuff in this today is world that's why the data scrapping has been playing a very important role if the important rules admin not completed by the data scrapping than the data scrapping is just 10 news list there are two many steps skipping if the slapping is not done then the skepping i

1. DATA SCRAPING Data scraping, or web scraping, is a process of importing data from websites into files or spreadsheets. It is used to extract data from the web, either for personal use by the scraping operator, or to reuse the data on other websites. There are numerous software applications for automating data scraping.

2. Uses Of Data Scraping Collect business intelligence to inform web content Determine prices for travel booking or comparison sites Find sales leads or conduct market research via public data sources Send product data from e Commerce sites to online shopping platforms like Google Shopping Data scraping has legitimate uses, but is often abused by bad actors. For example, data scraping is often used to harvest email addresses for the purpose of spamming or scamming. Scraping can also be used to retrieve copyrighted content from one website and automatically publish it on another website . Some countries prohibit the use of automated email harvesting techniques for commercial gain, and it is generally considered an unethical marketing practice.

3. Types of Data Scarping SCREEN SCRAPING Although the use of physical "dumb terminal" IBM 3270s is slowly diminishing, as more and more mainframe applications acquire Web interfaces, some Web applications merely continue to use the technique of  screen scraping  to capture old screens and transfer the data to modern front-ends . Screen scraping is normally associated with the programmatic collection of visual data from a source, instead of parsing data as in web scraping. Originally,  screen scraping  referred to the practice of reading text data from a computer display terminal's screen. This was generally done by reading the terminal's  memory through its auxiliary port, or by connecting the terminal output port of one computer system to an input port on another. The term screen scraping is also commonly used to refer to the bidirectional exchange of data. This could be the simple cases where the controlling program navigates through the user interface, or more complex scenarios where the controlling program is entering data into an interface meant to be used by a human.

4. Web scraping Web pages are built using text-based mark-up languages (HTML and XHTML), and frequently contain a wealth of useful data in text form. However, most web pages are designed for human end-users and not for ease of automated use. Because of this, tool kits that scrape web content were created. A web scraper is an  API or tool to extract data from a website .  Companies like Amazon AWS  and  Google  provide  web scraping  tools, services, and public data available free of cost to end-users. Newer forms of web scraping involve listening to data feeds from web servers. For example,  JSON  is commonly used as a transport storage mechanism between the client and the webserver Recently, companies have developed web scraping systems that rely on using techniques in DOM parsing, computer vision and natural language processing to simulate the human processing that occurs when viewing a webpage to automatically extract useful information . Large websites usually use defensive algorithms to protect their data from web scrapers and to limit the number of requests an IP or IP network may send. This has caused an ongoing battle between website developers and scraping developers

5. REPORT MINING Report mining  is the extraction of data from human-readable computer reports. Conventional data extraction requires a connection to a working source system, suitable  connectivity standards or an API, and usually complex querying. By using the source system's standard reporting options, and directing the output to a  spool file  instead of to a printer, static reports can be generated suitable for offline analysis via report mining.  This approach can avoid intensive  CPU  usage during business hours, can minimize   end-user   licence costs for ERP customers, and can offer very rapid prototyping and development of custom reports. Whereas data scraping and web scraping involve interacting with dynamic output, report mining involves extracting data from files in a human-readable format, such as HTML, PDF, or text. These can be easily generated from almost any system by intercepting the data feed to a printer. This approach can provide a quick and simple route to obtaining data without the need to program an API to the source system.

6. Data Scraping and Cyber security Data scraping tools are used by all sorts of businesses, not necessarily for  malicious purposes . These include marketing research and business intelligence, web content and design, and personalization. However, data scraping also poses challenges for many businesses, as it can be used to expose and misuse  sensitive data . The website being scraped might not be aware that their data is collected, or what is being collected. Likewise, a legitimate data scraper might not store the data securely, allowing   attackers  to access it. If malicious actors can access the data collected through web scraping, they can exploit it in  cyber attacks . For example, attackers can use scraped data to perform : Phishing attacks —attackers can leverage scraped data to sharpen their phishing techniques. They can find out which employees have the access permissions they want to target, or if someone is more susceptible to a phishing attack. If attackers can learn the identities of senior staff, they can carry out   spear phishing attacks , tailored to their target . Password cracking attacks —attackers can crack credentials to break through authentication protocols, even if the passwords aren’t leaked directly. They can study publicly available information about your employees to guess passwords based on personal details.

7. Data Scraping Techniques Here are a few techniques commonly used to scrape data from websites. In general, all  web scarping techniques retrieve content fro websites, process it using a scraping engine, and generate one or more data files with the extracted content. HTML Parsing HTML parsing involves the use of JavaScript to target a linear or nested HTML page. It is a powerful and fast method for extracting text and links (e.g. a nested link or email address), scraping screens and pulling resources. DOM Parsing The Document Object Model (DOM) defines the structure, style and content of an XML file. Scrapers typically use a DOM parser to view the structure of web pages in depth. DOM parsers can be used to access the nodes that contain information and scrape the web page with tools like XPath . For dynamically generated content, scrapers can embed web browsers like Firefox and Internet Explorer to extract whole web pages (or parts of them).

8. Vertical Aggregation Companies that use extensive computing power can create vertical aggregation platforms to target particular verticals. These are data harvesting platforms that can be run on the cloud and are used to automatically generate and monitor bots for certain verticals with minimal human intervention. Bots are generated according to the information required to each vertical, and their efficiency is determined by the quality of data they extract . Xpath XPath is short for XML Path Language, which is a query language for XML documents. XML documents have tree-like structures, so scrapers can use XPath to navigate through them by selecting nodes according to various parameters. A scraper may combine DOM parsing with XPath to extract whole web pages and publish them on a destination site . Google Sheets Google Sheets is a popular tool for data scraping. Scarpers can use the IMPORTXML function in Sheets to scrape from a website, which is useful if they want to extract a specific pattern or data from the website. This command also makes it possible to check if a website can be scraped or is protected.

9. How to Mitigate Web Scraping For content to be viewable, web content usually needs to be transferred to the machine of the website viewer. This means that any data the viewer can access is also accessible to a scraping bot. You can use the following methods to reduce the amount of data that can be scraped from your website . Rate Limit User Requests The rate of interaction for human visitors clicking through a website is relatively predictable. For example, it is impossible for a human to go through 100 web pages per second, while machines can make multiple simultaneous requests. The rate of requests can indicate the use of data scraping techniques that attempt to scrape your entire site in a short time. You can rate limit the number of requests an IP address can make within a particular time frame. This will protect your website from exploitation and significantly slow down the rate at which data scraping can occur . Mitigate High-Volume Requesters with CAPTCHAs Another way to slow down data scraping efforts is to apply CAPTCHAs . These require website visitors to complete a task that would be relatively easy for a human but prohibitively challenging for a machine. Even if a bot can get past the CAPTCHA once, it will not be able to do so across multiple instances. The drawback of CAPTCHA challenges is their potential negative impact on user experience.

10. Regularly Modify HTML Markup A data scraping bot needs consistent formatting to be able to traverse a website and parse useful information effectively. You can interrupt the workflow of a bot by modifying HTML markup elements on a regular basis. For example, you can nest HTML elements or change various markup aspects, which will make it more difficult to scrape consistently. Some websites implement randomized modifications whenever they are rendered, in order to protect their content. Alternatively, websites can modify their markup code less frequently, with the aim of preventing a longer-term data scraping effort . Embed Content in Media Objects This is a less popular method of  mitigation that involves media objects such as images . To extract data from image files, you need to use optical character recognition (OCR), as the content doesn’t exist as a string of characters. This makes the process of copying content much more complicated for data scrapers, but it can also be an obstacle to legitimate web users, who will not be able to copy content from the website and must instead retype or memorize it. However, the above methods are partial and do not guarantee protection against scraping. To fully protect your website, deploy a  bot protection solution  that detects scraping bots, and is able to block them before they connect to your website or web application.