closd computing 4th updated MODULE-4.pptx

it is a basic ppt regarding mod 4 cloud application development in cloud

1. CLOUD COMPUTING – CSE 2013 Module 4: Cloud Application Development in Cloud

2. Overview of topics PROGRAMMING MODEL FOR CLOUD COMPUTING SOFTWARE DEVELOPMENT IN CLOUD SERVICE CREATION ENVIRONMENTS TO DEVELOP CLOUD-BASED APPLICATION DEVELOPMENT ENVIRONMENTS FOR SERVICE DEVELOPMENT(DEMONSTRATION USING AWS CLOUD) DOCKERS & CONTAINERS

3. Programming Models for Cloud Computing Programming model is a specific way or a method or approach There are several programming models available in general , each having their own advantages and disadvantages Programming models form the base for any software or application development approach. The properties of these programming models decide their usage and their impact . As far as general programming models are concerned, these have continuously evolved.

4. Cont.. Each evolutionary step is characterized by certain changes to the existing methodology , and at each level, a  new functionality is added, which facilitates the programmers in one or several ways. As the technology is growing, so are the problems. One programming model cannot be the solution for all the problems , even with a large number of advancements. In these cases, a whole new programming model is developed for a specific set of problems or for general use. To address an issue, any of the two approaches can be used.

5. Cont.. Cloud computing is one of the most popular and extensively used technologies. As a technology, cloud is considered to be good and is popular because it allows users to primarily use computing, storage, and networking services on demand without spending for infrastructure (capital expenditure). An extensive research is going on in this segment. Many business organizations have started depending on cloud. The fast development of cloud was driven by its market-oriented nature . Several big companies  like Amazon, Google, and Microsoft have started using cloud extensively , and they intend to continue this trend.

6. Cont.. With the advent of cloud computing, there came a new shift in the computing era. All the applications are being migrated to cloud as the future seems to be fully dependent on it. The service architecture of cloud has three basic components called service models, namely Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Among these three, SaaS is the most important model where software is given as a service from cloud. This software is specifically designed for cloud platform, and for designing a software or application, you have to use a specific programming model .

7. Cloud has several properties that make it totally different from other conventional software applications, such as scalability, concurrency, multitenancy, and fault tolerance. There are generally two types of practices: Extending the existing programming models Completely new model specifically designed for cloud .

8. Extended Programming Models for Cloud The existing programming models that can be extended to a cloud platform. Not all programming models can be migrated to cloud. There are several properties of cloud that make it a different technology from others. Thus, certain changes have to be made to address the issues of the cloud. Only certain programming models that have some similarity to cloud or have a structure that can eventually support certain parameters like scalability, concurrency, and multitenancy can be extended. Out of these properties, scalability and concurrency are considered to be very important for cloud . Scalability is the ability of the system to scale itself up or down according to the number of users. This is one of the reasons behind the popularity of cloud. Thus, this property should be taken care of by any programming model.

9. EXTENDED PROGRAMMING MODEL MAPREDUCE CGL MAPREDUCE CLOUD HASKELL

10. MapReduce MapReduce is a programming framework that allows us to perform distributed and parallel processing on large data sets in a distributed environment.

11. MapReduce consists of two distinct tasks  Map - Mapping the Data Reduce – Reducing the Data

12. Working T he first is the map job, where a block of data is read and processed to produce key-value pairs as intermediate outputs. The output of a Mapper or map job (key-value pairs) is input to the Reducer. The reducer receives the key-value pair from multiple map jobs. Then, the reducer a ggregates those intermediate data tuples (intermediate key-value pair) into a smaller set of tuples or key-value pairs which is the final output.

13. A Word Count Example of MapReduce

14. PROCESS First, we divide the input into three splits as shown in the figure. This will distribute the work among all the map nodes . Then, we tokenize the words in each of the mappers and give a hardcoded value (1) to each of the tokens or words. The rationale behind giving a hardcoded value equal to 1 is that every word, in itself, will occur once. Now, a list of key-value pair will be created where the key is nothing but the individual words and value is one. So, for the first line (Dear Bear River) we have 3 key-value pairs — Dear, 1; Bear, 1; River, 1. The mapping process remains the same on all the nodes.

15. After the mapper phase, a partition process takes place where sorting and shuffling happen so that all the tuples with the same key are sent to the corresponding reducer. So, after the sorting and shuffling phase, each reducer will have a unique key and a list of values corresponding to that very key. For example, Bear, [1,1]; Car, [1,1,1].., etc. Now, each Reducer counts the values which are present in that list of values. As shown in the figure, reducer gets a list of values which is [1,1] for the key Bear. Then, it counts the number of ones in the very list and gives the final output as — Bear, 2. Finally, all the output key/value pairs are then collected and written in the output file.

16. Advantages of MapReduce 1. Parallel Processing 2. Data Locality

17. CGL-MAPREDUCE CGL-MapReduce is another type of MapReduce runtime and was developed by Ekanayake There are specific differences between CGLMapReduce and MapReduce. Unlike MapReduce, CGL-MapReduce uses streaming instead of a file system for all communications. This eliminates the overheads associated with file communication , and the intermediate results from the map functions are directly sent. The application was primarily created and tested for a large amount of scientific data and was compared with MapReduce

18. Diff: CGLMapReduce is different from MapReduce, the main difference being the avoidance of file system and usage of streaming.

19. PROCESS

20. WORKING COMPONENTS Map worker: A map worker is responsible for doing map operation. 2. Reduce worker: A reduce worker is responsible for doing reduce operation. 3. Content Dissemination Network: Content dissemination network handles all the communication between the components using the NaradaBroker , which was developed by Ekanayake et. 4.MRDriver : MRDriver is a master worker and controls the other workers based on the instructions by the user program.

21. steps involved in CGL-MapReduce

22. STAGE :1  INITIALIZATION STAGE The first step involves starting the MapReduce worker nodes and configuration of the MapReduce task. Configuration of the MapReduce is a onetime process and multiple copies of it can be reused. This is one of the improvements of CGL-MapReduce which facilitates efficient iterative MapReduce computations . These configured MapReduce are stored and executed upon the request of the user. Here, fixed data are given as input.

23. STAGE :2  MAP STAGE After the initialization step, MRDriver starts the map computation upon the instruction of the programmer. This is done by passing the variable data to the map tasks. This is relayed to workers for invoking configured map tasks. It also allows passing the results from one iteration to another. Finally, the map tasks are transferred directly to reduce workers using dissemination network.

24. STAGE :3  REDUCE STAGE As soon as all the map tasks are completed, they are transferred to reduce workers and these workers start executing tasks after they are initialized by the MRDriver . Output of the reduce function is directly sent to the user application

25. STAGE :4  COMBINE STAGE In this stage, all the results obtained in the reduce stage are combined.

26. STAGE :5  TERMINATION STAGE This is the final stage, and user program gives the command for termination. At this stage, all the workers are terminated

27. Key features Uses streaming for communication Supports parallelization Iterative in nature Can handle a large amount of data

28. CLOUD HASKELL Cloud Haskell is based on the Haskell functional programming language . These functional programming languages are function based and work like mathematical functions . Usually, the output here is based on the input value only . Functional programming consists of components that are often immutable . Here, in Cloud Haskell, the immutable components form the basis . Immutable components are the components whose states cannot be modified after they have been created. Cloud Haskell is a domain-specific language

29. Cont.. The programming model is completely based on message-passing interface which can be used for highly reliable real-time application. The main aim is to use concurrent programming paradigm. The Cloud consists of several applications and users who are independent and are large in numbers. Thus, to address the issue, concurrency has to be ensured for a large number of requests. Cloud Haskell provides the concurrency features with complete independence for the application

30. Concurrency This model is not designed to have a shared memory. As the model supports concurrency and these processes are isolated, fault tolerance is expected to an extent and the failure of one process would not affect others.

31. HASKELL  serialization Whenever a programmer tries to use the distributed code, he had to run the code in a remote system. This was not possible in Haskell versions. Thus, in this model, the developers introduce a mechanism that will automatically take care of this issue without extending the compiler. In simple terms, the serialization is taken care of with full abstraction so that the programmer does not know it. The model also takes care of failure, which is an important issue in cloud. There are many ways in which failure might affect the functioning of the system. As the number of systems connected and users increase, failure rate also increases.

32. Idempotency All the data are immutable by default and functions are idempotent. Idempotency here refers to the ability of the functions to restart from any points on its own without the use of any external entity such as a distributed database when the function is aborted due to hardware error.

33. FAULT TOLERENCE Usually, to confront partial failure, the whole system is restarted. This is considered as one of the inefficient solutions as when it comes to cloud the number of distributed nodes is high, and so restarting a system can prove to be very costly. Thus, to maintain fault tolerance, Erlang’s method is used. Even if a function fails because of certain reasons, it can be separately restarted without affecting the other parts and processes

34. Key features Fault tolerant Domain-specific language Uses concurrent programming paradigm No shared memory Idempotent Purely functionaL

35. SOFTWARE DEVELPOMENT IN CLOUD Traditional S/W Development vs the Current Scenarios Cloud Analyst AWS Saturn Cloud

36. Basics SaaS is a promising software delivery and business model in the information technology (IT) industry. The applications will be deployed by the SaaS provider in their managed or hosted infrastructure. The end users can access the hosted application as a service whenever there is a need. For using SaaS, the end users need to pay the full license fee . They can pay for what they used or consumed . To access the SaaS application, the customers need not install the software on their devices. It can be accessed from the service provider infrastructure using a simple web browser over the Internet.

37. Traditional S/w Delivery Model .In the traditional software delivery model, the relationship between the end user and the software is one to one and licensing based. The end users have to buy the software from the vendors by paying a huge licensing amount. Some heavyweight applications need a high computing power. So, the end users have to buy the required hardware also. So, the initial investment to use the software is high , and if you look at the usage, it will be very low.

38. To Overcome the Disadvantage To overcome this disadvantage of the traditional software delivery model, companies started developing the SaaS application which has one-to-many relationships with the end users and the software. SaaS follows the multitenant architecture , and it allows many customers to share a single instance of the software , popularly known as multitenancy . Because of its cost-effective nature , many customers started moving to SaaS applications rather than the traditional licensing-based software . SaaS applications benefit not only the end users but also the service providers

39. SaaS Is Different from Traditional Software SaaS provides web access to commercial software on pay-as-you-use basis. • SaaS applications are developed, deployed, and managed from a central location by the service provider. • It allows the same instance of the SaaS application to be shared by multiple customers or tenants. • Updates are performed by the service provider, not by the user . • SaaS applications allow service integration with other third-party services through application programming interfaces (APIs) provided by them

40. SaaS Benefits Pay per use Zero infrastructure Ease of access Automated updates Composite services Dynamic scaling

41. SaaS from IaaS and Managed Platform The SaaS providers can use the infrastructure provided by any IaaS provider The infrastructure provider may be a public or private IaaS provider. The public infrastructure will be chosen if the SaaS application does not require more security to the data. If the application needs more security and at the same time more resource utilization , they can choose the private IaaS model. Here, the multitenancy will be achieved at the infrastructure and application layers. The service providers have to manage their own development platform . Since the infrastructure is given by a third party, there is a possibility of vendor lock-in at the IaaS layer

42. Typical architecture of the SaaS application.

43. Pros Ensures high resource utilization at the infrastructure level. • Reduces the capital investment on the infrastructure. • Capital investment can be reduced

44. Cons Still there is additional overhead in maintaining the development platform. • Has to enable highly scalable and available features manually. • There is a possibility of vendor lock-in at the infrastructure layer.

45. Tools for Cloud Services Demonstrations

46. Cloud Analyst Cloud Analyst is a tool developed at the University of Melbourne whose goal is to support evaluation of social networks tools according to geographic distribution of users and data centers. In this tool, communities of users and data centers supporting the social networks are characterized and, based on their location; parameters such as user experience while using the social network application and load on the data center are obtained/logged

47. http://cloudsim-setup.blogspot.com/2013/01/running-and-using-cloud-analyst.html

48. CloudSim Architecture

49. The One Drive Link to access the Steps to Run CloudAnalyst Cloudanalyst.pdf https://presidencyuniversityin-my.sharepoint.com/:b:/g/personal/gopalkirshna_shyam_presidencyuniversity_in/EQ-ZpiWTC5BNsljJQvYoZSABo1fIovr3aYfxfRU9brQQbg?e=bS4Q0u

50. AWS Cloud

51. AWS Cloud Online Link https://signin.aws.amazon.com/signin?redirect_uri=https%3A%2F%2Fconsole.aws.amazon.com%2Fec2%2Fv2%2Fhome%3Frefid%3Dfb8718a7-d9f7-4e07-9fc5-b85de26b4178%26state%3DhashArgs%2523%26isauthcode%3Dtrue&client_id=arn%3Aaws%3Aiam%3A%3A015428540659%3Auser%2Fec2&forceMobileApp=0&code_challenge=3ui6zNA0lwTp9FdyE6ifeTh6-KQ6c5vXuZir5LucyS8&code_challenge_method=SHA-256

52. Saturn Cloud

53. Saturn Cloud runs compute with exactly the hardware, software, and Git repositories you want. 

54. Introduction to Docker

55. Static website Web frontend User DB Queue Analytics DB Background workers API endpoint nginx 1.5 + modsecurity + openssl + bootstrap 2 postgresql + pgv8 + v8 hadoop + hive + thrift + OpenJDK Ruby + Rails + sass + Unicorn Redis + redis -sentinel Python 3.0 + celery + pyredis + libcurl + ffmpeg + libopencv + nodejs + phantomjs Python 2.7 + Flask + pyredis + celery + psycopg + postgresql -client Development VM QA server Public Cloud Disaster recovery Contributor’s laptop Production Servers The Challenge Multiplicity of Stacks Multiplicity of hardware environments Production Cluster Customer Data Center Do services and apps interact appropriately? Can I migrate smoothly and quickly?

56. Multiplicity of Goods Multipilicity of methods for transporting/storing Do I worry about how goods interact (e.g. coffee beans next to spices) Can I transport quickly and smoothly (e.g. from boat to train to truck) Cargo Transport Pre-1960

57. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Also a matrix from hell

58. Multiplicity of Goods Multiplicity of methods for transporting/storing Do I worry about how goods interact (e.g. coffee beans next to spices) Can I transport quickly and smoothly (e.g. from boat to train to truck) Solution: Intermodal Shipping Container …in between, can be loaded and unloaded, stacked, transported efficiently over long distances, and transferred from one mode of transport to another A standard container that is loaded with virtually any goods, and stays sealed until it reaches final delivery.

59. Static website Web frontend User DB Queue Analytics DB Development VM QA server Public Cloud Contributor’s laptop Docker is a shipping container system for code Multiplicity of Stacks Multiplicity of hardware environments Production Cluster Customer Data Center Do services and apps interact appropriately? Can I migrate smoothly and quickly …that can be manipulated using standard operations and run consistently on virtually any hardware platform An engine that enables any payload to be encapsulated as a lightweight, portable, self-sufficient container…

60. Static website Web frontend Background workers User DB Analytics DB Queue Development VM QA Server Single Prod Server Onsite Cluster Public Cloud Contributor’s laptop Customer Servers Docker eliminates the matrix from Hell

61. Why Developers Care Build once…(finally) run anywhere* A clean, safe, hygienic and portable runtime environment for your app. No worries about missing dependencies, packages and other pain points during subsequent deployments. Run each app in its own isolated container, so you can run various versions of libraries and other dependencies for each app without worrying Automate testing, integration, packaging…anything you can script Reduce/eliminate concerns about compatibility on different platforms, either your own or your customers. Cheap, zero-penalty containers to deploy services? A VM without the overhead of a VM? Instant replay and reset of image snapshots? That’s the power of Docker * With the 0.7 release, we support any x86 server running a modern Linux kernel (3.2+ generally. 2.6.32+ for RHEL 6.5+, Fedora, & related)

62. Why Devops Cares? Configure once…run anything Make the entire lifecycle more efficient, consistent, and repeatable Increase the quality of code produced by developers. Eliminate inconsistencies between development, test, production, and customer environments Support segregation of duties Significantly improves the speed and reliability of continuous deployment and continuous integration systems Because the containers are so lightweight, address significant performance, costs, deployment, and portability issues normally associated with VMs

63. Why it works—separation of concerns Dan the Developer Worries about what’s “inside” the container His code His Libraries His Package Manager His Apps His Data All Linux servers look the same Oscar the Ops Guy Worries about what’s “outside” the container Logging Remote access Monitoring Network config All containers start, stop, copy, attach, migrate, etc. the same way

64. More technical explanation High Level—It’s a lightweight VM Own process space Own network interface Can run stuff as root Can have its own / sbin / init (different from host) <<machine container>> Low Level—It’s chroot on steroids Can also not have its own / sbin / init Container=isolated processes Share kernel with host No device emulation (neither HVM nor PV) from host) <<application container>> Run everywhere Regardless of kernel version (2.6.32+) Regardless of host distro Physical or virtual, cloud or not Container and host architecture must match* Run anything If it can run on the host, it can run in the container i.e. if it can run on a Linux kernel, it can run WHY WHAT

65. App A Containers vs. VMs Hypervisor (Type 2) Host OS Server Guest OS Bins/ Libs App A’ Guest OS Bins/ Libs App B Guest OS Bins/ Libs App A’ Docker Host OS Server Bins/Libs App A Bins/Libs App B App B’ App B’ App B’ VM Container Containers are isolated, but share OS and, where appropriate, bins/libraries Guest OS Guest OS …result is significantly faster deployment, much less overhead, easier migration, faster restart

66. Why are Docker containers lightweight? Bins/ Libs App A Original App (No OS to take up space, resources, or require restart) App Δ Bins/ App A Bins/ Libs App A’ Guest OS Bins/ Libs Modified App Copy on write capabilities allow us to only save the diffs Between container A and container A’ VMs Every app, every copy of an app, and every slight modification of the app requires a new virtual server App A Guest OS Bins/ Libs Copy of App No OS. Can Share bins/libs App A Guest OS Guest OS VMs Containers

67. What are the basics of the Docker system? Source Code Repository Docker file For A Docker Engine Docker Container Image Registry Build Docker Host 2 OS (Linux) Container A Container B Container C Container A Push Search Pull Run Host 1 OS (Linux)

68. Changes and Updates Docker Engine Docker Container Image Registry Docker Engine Push Update Bins/ Libs App A App Δ Bins/ Base Container Image Host is now running A’’ Container Mod A’’ App Δ Bins/ Bins/ Libs App A Bins/ Bins/ Libs App A’’ Host running A wants to upgrade to A’’. Requests update. Gets only diffs Container Mod A’

69. Ecosystem Support Operating systems Virtually any distribution with a 2.6.32+ kernel Red Hat/ Docker collaboration to make work across RHEL 6.4+, Fedora, and other members of the family (2.6.32 +) CoreOS —Small core OS purpose built with Docker OpenStack Docker integration into NOVA (& compatibility with Glance, Horizon, etc.) accepted for Havana release Private PaaS OpenShift Solum (Rackspace, OpenStack ) Other TBA Public PaaS Deis , Voxoz , Cocaine ( Yandex ), Baidu PaaS Public IaaS Native support in Rackspace, Digital Ocean,+++ AMI (or equivalent) available for AWS & other DevOps Tools Integrations with Chef, Puppet, Jenkins, Travis, Salt, Ansible +++ Orchestration tools Mesos , Heat, ++ Shipyard & others purpose built for Docker Applications 1000’s of Dockerized applications available at index.docker.io

70. Use Cases Ted Dziuba on the Use of Docker for Continuous Integration at Ebay Now https://speakerdeck.com/teddziuba/docker-at-ebay http://www.youtube.com/watch?feature=player_embedded&v=0Hi0W4gX--4 Sasha Klizhentas on use of Docker at Mailgun /Rackspace http://www.youtube.com/watch?feature=player_embedded&v=CMC3xdAo9RI Sebastien Pahl on use of Docker at CloudFlare http://www.youtube.com/watch?feature=player_embedded&v=-Lj3jt_-3r0 Cambridge HealthCare http://blog.howareyou.com/post/62157486858/continuous-delivery-with-docker-and-jenkins-part-i Red Hat Openshift and Docker https://www.openshift.com/blogs/technical-thoughts-on-openshift-and-docker

71. Use Cases—From Our Community Use Case Examples Link Clusters Building a MongoDB cluster using docker http://bit.ly/1acbjZf Production Quality MongoDB Setup with Docker http://bit.ly/15CaiHb Wildfly cluster using Docker on Fedora http://bit.ly/1bClX0O Build your own PaaS OpenSource PaaS built on Docker, Chef, and Heroku Buildpacks http://deis.io Web Based Environment for Instruction JiffyLab – web based environment for the instruction, or lightweight use of, Python and UNIX shell http://bit.ly/12oaj2K Easy Application Deployment     Deploy Java Apps With Docker = Awesome http://bit.ly/11BCvvu How to put your development environment on docker http://bit.ly/1b4XtJ3 Running Drupal on Docker http://bit.ly/15MJS6B Installing Redis on Docker http://bit.ly/16EWOKh Create Secure Sandboxes Docker makes creating secure sandboxes easier than ever http://bit.ly/13mZGJH Create your own SaaS Memcached as a Service http://bit.ly/11nL8vh Automated Application Deployment Multi-cloud Deployment with Docker http://bit.ly/1bF3CN6 Continuous Integration and Deployment Next Generation Continuous Integration & Deployment with dotCloud’s Docker and Strider http://bit.ly/ZwTfoy Testing Salt States Rapidly With Docker http://bit.ly/1eFBtcm Lightweight Desktop Virtualization Docker Desktop: Your Desktop Over SSH Running Inside Of A Docker Container   http://bit.ly/14RYL6x

72. Docker Futures* Docker 0.7 (current release) Fedora compatibility Reduce kernel dependencies Device mapper Container linking Docker 0.8 (Dec) Shrink and stabilize Core Provide stable, pluggable API RHEL compatibility Nested containers Beam: Introspection API based on Redis expand snapshot management features for data volumes We will consider this “production ready” Docker 0.9 (Jan) Docker 1.0 (Feb) We will offer support for this product Docker 0.1-0.6 AUFS Docker 0.8+ LXC Device Mapper LXC LIBVIRT JAILS SELinux Service Discover * We shoot for time based releases (1x/5wks), features are targeted, but not guaranteed for particular releases

73. www.docker.io

74. QUERIES?