DevOps DevSecOps Based on Training Materials

DevOps and DevSecOps

1. DevOps and DevSecOps Based on JanBask Training Material

2. Where did DevOps Come from? ESM (Enterprise Systems Management):  People involved in the initial phases of DevOps are system administrators. brought the key ESM practices to DevOps like configuration management, automated provisioning, system monitoring, and the toolchain approach, etc. Agile Development : outgrowth of the agile. extending the Agile principles beyond boundaries of the code to the entire delivered services. “When you are going agile without DevOps, it is like racing with a tractor instead of a car. You can do laps, but it will not move faster, and ultimately you are going to waste a lot of fuel without having any fun.”

3. What is DevOps? DevOps word in itself is a combination of two words one is Development and other is Operations. neither an application nor a tool; just a culture to promote development and Operation process collaboratively. the speed to deliver applications and services has increased. DevOps enables organizations to serve their customers strongly and better in the market. DevOps is the process of alignment of IT and development operations with better and improved communication.

4. What Problems led to the creation of DevOps? Before DevOps, operation and development teams were working in an isolated environment. Testing and Deployment activities mostly were performed in an isolated manner after design-build step took more time than actual project completion time. Team members usually spend a large amount of time in deploying, testing, designing, and building the projects Human production errors were deployed during manual code conduction. Operations and coding teams generally had different timelines and did not have proper synchronization that results in further delay.

5. How is DevOps different from Traditional IT? Traditional IT DevOps Once the order for new servers is placed, the development team starts working on testing. The development team has to continue with heavy paperwork as required by enterprises to deploy the infrastructure. Once the order for new servers is placed, the development team and operations team start the paperwork to set up new servers that result in better visibility of infrastructure equipment. Projections about failover, data center locations, redundancy, and storage requirements are not clear as no inputs are available from the development team even if they have the depth knowledge of the application. Projections about failover, data center locations, redundancy, and storage requirements are 100 percent clear because of accurate inputs given from the development team. In old software development processes, the operations team has no idea of the progress of the development team. Operation team has to prepare a monitoring plan as per their own understanding. In DevOps, the operations team have a complete idea of the progress of development. Operations team and development team work together to develop a monitoring plan that caters to the current business, and IT needs. Before go-live, the load testing may crash the application, and the release may get delayed. It affects the overall cost of the project and project delivery deadline. Before go-live, the load testing makes the application a little slow. The development team quickly fixes bottlenecks, and the application is released on time.

6. 3 Pillars of DevOps Infrastructure Automation Continuous Delivery Reliability Engineering

7. Infrastructure Automation Automate Everything Infrastructure provisioning Application Deployment Runtime Orchestration Model Driven Automation

8. Infrastructure Automation Tooling Infrastructure Models - AWS Cloudformation , Terraform , Azure ARM Templates, Ubuntu Juju Hardware Provisioning - Packer, Foreman, MaaS , Cobbler, Crowbar, Digital Rebar Configuration Management - Puppet, Chef, Ansible, Salt, CFEngine Integration Testing - rspec , serverspec Orchestration - Rundeck , Ansible, Kubernetes (for docker)

9. Signs that you need DevOps The development team is not able to detect software defects at the early age of its development Agile methods are used to speed up the software development process, but as soon as the application goes to production department all methods become ineffective Testing and development team members are not able to access resources timely and so the development process delays You are not able to identify the exact problems of development, testing, and production department Simple human errors are often creating hurdles during the development and deployment process. Once the app is in production, developers think that their job is over. At the time of the problem, both development and operation teams start blaming each other.

10. DevOps Features Predictability : DevOps decreases the failure rate of new product releases. Maintainability:  The process improves the overall recovery rate at the time of the release event. Improved Quality : DevOps improves the quality of product development by incorporating infrastructure issues. Lower Risk : Security aspects are incorporated in SDLC, and the number of defects gets decreased across the product Cost Efficient : Cost efficiency is improved due to DevOps that is always an aspiration of every business organization. Stability:  DevOps implementation offers a stable and secure operational state. Streamlined Delivery Process : As DevOps provides streamlined software delivery, marketing effort is reduced up to 50%.

11. What are the features of DevOps Implementation “DevOps is not a goal but a never-ending process of continual improvement.” The DevOps offer continuous integration and continuous delivery. It makes the product delivery cycle quicker, and enterprises become able to launch the software timely without compromising its quality.

12. DevOps Lifecycle Phases and Measures As per DevOps culture, a group of Engineers is responsible for each stage of DevOps application development,

13. DevOps Phases Development development process is broken down into small steps or development cycles Testing Selenium like testing tools is used to speed up the overall testing process by quick identification of errors and fixing the bugs. Integration New functionalities are integrated with the prevailing code, and testing of new code takes place. Continuous integration and testing help in the continuous development process. Deployment Continuous deployment is the part of DevOps lifecycle. Monitoring Inappropriate system behavior is managed by monitoring.

14. What is DevSecOps

15. DevSecOps Effort to strive for “Secure by Default” Integrate Security in Tools Create Security as a code Culture Promote cross skilling

16. Why do we need DevSecOps DevOps moves at a rapid pace Traditional Security just cannot keep pace Security as part of the process is the only way to ensure safety Security integrated into development, deployment and infrastructure is the need of DevSecOps

17. Traditional Security

18. Shifting left saves cost and time

19. How do we do DevSecOps DevSecOps is Automation + Cultural Changes Integrate security into your DevOps Pipeline Enable cultural changes to embrace DevSecOps

20. Injecting Sec in DevOps

21. A Sample Implementation of DevSecOps pipeline

22. Tools of Trade

23. Tools of Trade

24. Cultural Aspects Automation alone will not solve the problems Focus on collaboration and inclusive culture Encourage security mindset specially if it's outside sec team Build allies (security champions) in company Avoid Blame Game

25. Key Point Security is everyone responsibility • Embrace security as an integral part of the process, use feedback to refine the process • DevSecOps is not a one size fit all: your mileage will vary

26. Security Champion Bridge between Dev, Sec and Ops teams Build Security Champions Single Person per team Everyone provided with similar cross skilling opportunities Incentivize other teams to collaborate with Sec team Internal Bug bounties Sponsor Interactions (Parties / get-togethers) Sponsor cross skilling trainings for other teams

27. Case Study trove of more  than 24 million financial and banking documents, representing tens of thousands of loans and mortgages from some of the biggest banks in the U.S., has been found online after a server security lapse. The server, running an Elasticsearch database, had more than a decade’s worth of data, containing loan and mortgage agreements, repayment schedules and other highly sensitive financial and tax documents that reveal an intimate insight into a person’s financial life. But it wasn’t protected with a password, allowing anyone to access and read the massive cache of documents. It’s believed that the database was only exposed for two weeks — but long enough for independent security researcher  Bob Diachenko  to find the data. At first glance, it wasn’t immediately known who owned the data. After we inquired with several banks whose customers information was found on the server, the database was shut down on January 15. Prevention: Recurring Asset Inventory and Automated Assessments

28. Case Study Top defense contractor  Booz Allen Hamilton  leaks 60,000 files, including employee security credentials and passwords to a US government system. Verizon partner leaks personal records of  over 14 million Verizon customers , including names, addresses, account details, and for some victims — account PINs. An AWS S3 server leaked the personal details of  WWE fans  who registered on the company's sites. 3,065,805 users were exposed. Another AWS S3 bucket leaked the personal details of  over 198 million American voters . The database contained information from three data mining companies known to be associated with the Republican Party.   Another S3 database  left exposed only leaked the personal  details of job applications  that had Top Secret government clearance.   Dow Jones , the parent company of the Wall Street Journal, leaked the personal details of 2.2 million customers. Omaha-based voting machine firm Election Systems & Software (ES&S) left a database exposed online that contained the personal records of  1.8 million Chicago voters . Security researchers discovered a Verizon AWS S3 bucket containing over 100 MB of data about the  company's internal system  named Distributed Vision Services (DVS), used for billing operations. An  auto-tracking company  leaked over a half of a million records with logins/passwords, emails, VIN (vehicle identification number), IMEI numbers of GPS devices and other data that is collected on their devices, customers and auto dealerships. Prevention: Continuous monitoring and review of cloud assets and config https://www.bleepingcomputer.com/news/security/7-percent-of-all-amazon-s3-servers-are-exposed-explaining-recent-surge-of-data-leaks/

29. Case Study Prevention: Patching and Continuous monitoring of Assets

30. Security Threat Modeling

31. Common Types of Attack Connection Fails Organizational Attacks Restricted Data Accidental Breaches in Security Automated Attacks Hackers Viruses, Trojan Horses, and Worms Denial of Service (DoS) DoS

32. Types of Threats Spoofed packets, etc. Buffer overflows, illicit paths, etc. SQL injection, XSS, input tampering, etc. Network Host Application Threats against the network Threats against the host Threats against the application

33. Threats Against the Network Threat Examples Information gathering Port scanning Using trace routing to detect network topologies Using broadcast requests to enumerate subnet hosts Eavesdropping Using packet sniffers to steal passwords Denial of service (DoS) SYN floods ICMP echo request floods Malformed packets Spoofing Packets with spoofed source addresses http://msdn.microsoft.com/library/en-us/dnnetsec/html/THCMCh15.asp? frame=true#c15618429_004

34. Threats Against the Host Threat Examples Arbitrary code execution Buffer overflows in ISAPI DLLs (e.g., MS01-033) Directory traversal attacks (MS00-078) File disclosure Malformed HTR requests (MS01-031) Virtualized UNC share vulnerability (MS00-019) Denial of service (DoS) Malformed SMTP requests (MS02-012) Malformed WebDAV requests (MS01-016) Malformed URLs (MS01-012) Brute-force file uploads Unauthorized access Resources with insufficiently restrictive ACLs Spoofing with stolen login credentials Exploitation of open ports and protocols Using NetBIOS and SMB to enumerate hosts Connecting remotely to SQL Server

35. Threats Against the Application Threat Examples SQL injection Including a DROP TABLE command in text typed into an input field Cross-site scripting Using malicious client-side script to steal cookies Hidden-field tampering Maliciously changing the value of a hidden field Eavesdropping Using a packet sniffer to steal passwords and cookies from traffic on unencrypted connections Session hijacking Using a stolen session ID cookie to access someone else's session state Identity spoofing Using a stolen forms authentication cookie to pose as another user Information disclosure Allowing client to see a stack trace when an unhandled exception occurs

36. Threat Modeling Structured approach to identifying, quantifying, and addressing threats Essential part of development process Just like specing and designing Just like coding and testing

37. The Threat Modeling Process Identify assets Document architecture Decompose application Identify threats Document threats Rate threats 1 2 3 4 5 6

38. Identifying Assets What is it that you want to protect? Private data (e.g., customer list) Proprietary data (e.g., intellectual property) Potentially injurious data (e.g., credit card numbers, decryption keys) These also count as "assets" Integrity of back-end databases Integrity of the Web pages (no defacement) Integrity of other machines on the network Availability of the application 1

39. Documenting Architecture Define what the app does and how it's used Users view pages with catalog items Users perform searches for catalog items Users add items to shopping carts Users check out Diagram the application Show subsystems Show data flow List assets 2

40. Example Bob Alice Bill Asset #4 Asset #1 Asset #2 Asset #3 Asset #5 Asset #6 IIS ASP.NET Web Server Login State Main Database Server Firewall

41. Decomposing the App Refine the architecture diagram Show authentication mechanisms Show authorization mechanisms Show technologies (e.g., DPAPI) Diagram trust boundaries Identify entry points Begin to think like an attacker Where are my vulnerabilities? What am I going to do about them? 3

42. Example Bob Alice Bill IIS ASP.NET Web Server Database Server Trust Forms Authentication URL Authorization DPAPI Windows Authentication Firewall Login State Main

43. Identifying Threats Method #1: Threat lists Start with laundry list of possible threats Identify the threats that apply to your app Method #2: STRIDE Categorized list of threat types Identify threats by type/category Optionally draw threat trees Root nodes represent attacker's goals Trees help identify threat conditions 4

44. STRIDE S T R I D Tampering Repudiation Information disclosure Denial of service Can an attacker gain access using a false identity? Can an attacker modify data as it flows through the application? If an attacker denies doing something, can we prove he did it? Can an attacker gain access to private or potentially injurious data? Can an attacker crash or reduce the availiability of the system? E Elevation of privilege Can an attacker assume the identity of a privileged user? Spoofing

45. Threat Trees Theft of Auth Cookies Obtain auth cookie to spoof identity Unencrypted Connection Cookies travel over unencrypted HTTP Eavesdropping Attacker uses sniffer to monitor HTTP traffic Cross-Site Scripting Attacker possesses means and knowledge XSS Vulnerability Application is vulnerable to XSS attacks OR AND AND

46. Documenting Threats Theft of Auth Cookies by Eavesdropping on Connection Threat target Connections between browsers and Web server Risk Attack techniques Attacker uses sniffer to monitor traffic Countermeasures Use SSL/TLS to encrypt traffic Document threats using a template Theft of Auth Cookies via Cross-Site Scripting Threat target Vulnerable application code Risk Attack techniques Attacker sends e-mail with malicious link to users Countermeasures Validate input; HTML-encode output 5

47. Rating Threats Simple model DREAD model Greater granularization of threat potential Rates (prioritizes) each threat on scale of 1-15 Developed and widely used by Microsoft Risk = Probability * Damage Potential 1-10 Scale 1 = Least probable 10 = Most probable 1-10 Scale 1 = Least damage 10 = Most damage 6

48. DREAD D R E A D Reproducibility Exploitability Affected users Discoverability What are the consequences of a successful exploit? Would an exploit work every time or only under certain circumstances? How skilled must an attacker be to exploit the vulnerability? How many users would be affected by a successful exploit? How likely is it that an attacker will know the vulnerability exists? Damage potential

49. Example Threat D R E A D Sum Auth cookie theft (eavesdropping) 3 2 3 2 3 13 Auth cookie theft (XSS) 3 2 2 2 3 12 Potential for damage is high (spoofed identities, etc.) Cookie can be stolen any time, but is only useful until expired Anybody can run a packet sniffer; XSS attacks require moderate skill All users could be affected, but in reality most won't click malicious links Easy to discover: just type a <script> block into a field Prioritized Risks

50. Summary Without threat modelling, protecting yourself is like “shooting in the dark” You need expertise in understanding most common attacks – read security bulletins Developers must learn and use secure coding practices Learn some crypto too Assume you are vulnerable, prove you are not

51. References http://msdn.microsoft.com/security/securecode/threatmodeling/default.aspx http://sec.cs.kent.ac.uk/cms2004/Program/CMS2004final/p4a6.pdf http://cpd.ogi.edu/seminars04/hickmanthreatmodeling.pdf

52. Docker Security Underlying Technology of Docker Name Spaces namespaces provide the isolated workspace called the container. When you run a container, Docker creates a set of namespaces for that container. These namespaces provide a layer of isolation. Each aspect of a container runs in a separate namespace and its access is limited to that namespace. The pid namespace: Process isolation (PID: Process ID). The net namespace: Managing network interfaces (NET: Networking). The ipc namespace: Managing access to IPC resources (IPC: InterProcess Communication). The mnt namespace: Managing filesystem mount points (MNT: Mount). The uts namespace: Isolating kernel and version identifiers. (UTS: Unix Timesharing System).

53. Underlying Technologies of Docker Control Groups A cgroup limits an application to a specific set of resources. Control groups allow Docker Engine to share available hardware resources to containers and optionally enforce limits and constraints. For example, you can limit the memory available to a specific container. Union File Systems Union file systems, or UnionFS , are file systems that operate by creating layers, making them very lightweight and fast. Docker Engine uses UnionFS to provide the building blocks for containers. Docker Engine can use multiple UnionFS variants, including AUFS, btrfs , vfs , and DeviceMapper . Docker Engine combines the namespaces, control groups, and UnionFS into a wrapper called a container format

54. Docker Security Some of the common security problems face with docker Kernel exploits:  Since the host’s kernel is shared in the container, a compromised container can attack the entire host. Container breakouts:  Caused when the user is able to escape the container namespace and interact with other processes on the host. Denial-of-service attacks:  Occur when some containers take up enough resources to hamper the functioning of other applications. Poisoned images:  Caused when an untrusted image is being run and a hacker is able to access application data and, potentially, the host itself.

55. Docker Security Tips Use a Third-Party Security Tool Docker allows you to use containers from untrusted public repositories, which increases the need to scrutinize whether the container was created securely and whether it is free of any corrupt or malicious files. Tools: Anchore -- https://github.com/anchore/anchore-engine Clair -- https://github.com/quay/clair Dagda -- https://github.com/eliasgranderubio/dagda image security scanning is a process for finding security vulnerabilities within your Docker image files.  image security scanning is one critical way to find security flaws that could lead to a breach within a containerized application, it's important to note that security scanning by no means provides full security coverage.  image scanning tools check public security vulnerabilities databases if you include open source code in a container by importing it as a tarball instead of using a package from a public repository, your image scanner probably won't be able to scan that code,

56. Docker Security Tips Manage Vulnerability have a sound vulnerability management program that has multiple checks throughout the container lifecycle. Vulnerability management should incorporate quality gates to detect access issues and weaknesses for a potential exploit from dev-to-production environments. Tools Docker-bench-security -- https://github.com/docker/docker-bench-security OpemSCAP workbench’s oscap -docker utility Banyanops Collector - https://github.com/banyanops/collector

57. Docker Security Tips Monitor and Audit Container Activity It is vital to monitor the container ecosystem and detect suspicious activity. Container monitoring activities provide real-time reports that can help you react promptly to a security breach. Tools Sysdig Falco -- https://github.com/falcosecurity/falco Use Falco to monitor when a shell runs in a container, where a container has been mounted, unexpected reads of sensitive files, outbound network attempts, or other suspicious calls. Dagda https://github.com/eliasgranderubio/dagda You can run it remotely, or continually call it to monitor  active Docker containers . Cilium - https://github.com/cilium/cilium CoreOS developed Cilium in response to the volatile lifecycles of modern microservices development and quick container deployment. 

58. Docker Security Tips Enable Docker Content Trust Docker Content Trust  is a new feature incorporated into Docker 1.8. It is disabled by default, but once enabled, allows you to verify the integrity, authenticity, and publication date of all Docker images from the Docker Hub Registry. Use Docker Bench for Security You should consider  Docker Bench for Security  as your must-use script. Once the script is run, you will notice a lot of information regarding configuration best practices for deploying Docker containers that can be used to further secure your Docker server and containers.