5. rodando containers docker na aws

Containers para Software! A mais nova revolução, trazida ao mundo pela Dockers, rodando hoje na AWS. Venha conhecer esta inovadora e revolucionária tecnologia que vai mudar a forma como você desenvolve e implementa software.

1. São Paulo

2. Rodandocontainers Dockerna
AWS
Rafael Felix Correa
Consultant, AWS Professional Services

3. Agenda
IntroduçãoaoDocker
Formasde rodarcontainers DockernaAWS
Casode sucesso–MercadoLibre
Perguntas

4. Static website
Web frontend
User DB
QueueAnalytics DB
Background workers
API endpoint
nginx1.5 + modsecurity+ openssl+
bootstrap 2
postgresql+ pgv8 + v8
hadoop+ hive + thrift + OpenJDK
Ruby + Rails + sass + Unicorn
Redis+ redis-sentinel
Python 3.0 + celery + pyredis+ libcurl+ ffmpeg+
libopencv+ nodejs+ phantomjs
Python 2.7 + Flask + pyredis+ celery + psycopg+
postgresql-client
VM de desenvolvimento
Servidorde QA
Public Cloud
Disaster recovery
Notebook do funcionário
Servidoresde produção
O desafio
Variedade
de
Stacks
Variedade
de
ambientes
Cluster de produção
Data Center do Cliente
Serviços
e
aplicaçõesinteragem
apropriadamente
?
Consigo
migrar
rápido
e
sem
problemas
?

5. A matrizdo inferno
Static website
Web frontend
Background workers
User DB
Analytics DB
Queue
VM de
desenvolvim
ento
Servidorde
QA
Servidor
únicode
Prod
Cluster
interno
Public Cloud
Notebook do
funcionário
Servidores
do cliente
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?

6. Variedade
de
bens
Variedade
de
meios
de
transporte
e
armazenamento
Devo
me
preocupar
sobre
como
os
bens
interagem
?
(
exemplo
:
grãos
de
café
próximos
às
especiarias
)
Consigo
transportar
rápido
sem
problemas
?
(
exemplo
: do
navio
para
o
trem
para
o
caminhão
)
Transportede cargaantes de 1960

7. ? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
Outramatrizdo inferno

8. Variedade
de
bens
Variedade
de
meios
de
transporte
e
armazenamento
Devo
me
preocupar
sobre
como
os
bens
interagem
?
(
exemplo
:
grãos
de café
próximos
à
especiarias
)
Consigo
transportar
rápido
e
sem
problemas
? (
exemplo
:
do
navio
para
o
trem
para
o
caminhão
)
Solução: Container padronizadoparatransporte
…no caminho, podeser
carregado, descarregado,
empilhado, transportado
porgrandesdistânciase
transferidode umaforma
de transporteparaoutra
Um container padrão
carregadocom virtualmente
qualquerbem, se mantém
seladoatéchegarno destino
final.

9. Static website Web frontend User DB Queue Analytics DB
VM de
desenvolvimen
to
Servidorde QA
Public Cloud
Notebook
do
funcionário
Dockeréum sistemade enviode container para
código
Variedade
de
Stacks
Variedade
de
ambientes
Cluster de
produção
Data Center
do Cliente
Serviços
e
aplicações
interagem
apropriadamente
?
Consigo
migrar
rápido
e
sem
problemas
?
…quepodesermanipulado
e operadode forma
consistenteemvirtualmente
qualquerplataformade
hardware
Uma engine quepermite
qualquerconteúdoser
encapsuladocomoum
container leve, portável
e auto-suficiente…

10. Static website
Web frontend
Background workers
User DB
Analytics DB
Queue
VM de
desenvolvim
ento
Servidorde
QA
Servidor
únicode
Prod
Cluster
interno
Public Cloud
Notebook do
funcionário
Servidores
do cliente
Dockereliminaa matrizdo inferno

11. Quaisosbenefícios?
Negócio:
•Reduçãode custos
•Velocidade(Time to Market)
•Confiançae previsibilidadede entrega
Áreade TI:
•Separaçãode responsabilidadesmaisclara
•Portabilidadede aplicações
•Liberdadeparadevse times de produtosemperdero controle
•Aumentode velocidade(containers sãoleves)
•Aumentoda confiança(containers sãopadronizados)
•Usoeficientede recursosemescala(usandoum cluster manager)

12. O queéum container?
?!?
•Conjuntode featuresdo kernel do Linux queisolamprocessos, user
ids, memória, I/O e network
•“chrootcom esteróides!”
•Alocaçãode recursosgranular
•Namespaces, Cgroups
•Essasfeaturesexisteme tem sidousadasháanos(Solaris “Zones”)

13. App
A
Hypervisor
Host OS
Server
Guest
OS
Bins/
Libs
App
A’
Guest
OS
Bins/
Libs
App
B
Guest
OS
App A’
Docker
Host OS
Server
Bins/Libs
App A
Bins/Libs
App BApp B’App B’App B’
VM
Container
Containers sãoisolados,
mas compartilhamo SO e,
quandose aplica, bins/libs
Guest
OS
Guest
OS
…o quefazcom queoscontainers
possamserconsideradoscomo“VMs
enxutas”.
Bins/
Libs
Issopareceumamáquinavirtual (VM)!

14. SobreDocker
•Dockeréumaplataformacompletaparase construire
rodarcontainers
•Foidisponibilizadocomoopen source emMarçode
2013 peladotCloud(atualmenteDocker, Inc.)
•Incluium CLI, ferramentasparacriaçãode imagensde
containers (Dockerfiles), e um repositórioparahospedar
containers

15. Como o Dockerfunciona

16. Dockerfile
Imagem
dehttp://crosbymichael.com/dock
erfile-best-practices.html

17. Containers naAWS
•AWS éum complementonatural aousode containers
porsuavastagamade serviçosescaláveisde
infraestrutura, ondeseuscontainers podemrodar.
•AWS Elastic Beanstalk, AWS OpsWorks, e Amazon EC2
Container Service provémsuporteintegradoaoDocker.
•Oportunidadeparausarcontainers emDev, Test, e
Produção.

18. AWS Elastic Beanstalk
•AWS Elastic Beanstalk éumacamadade gerenciamentoparaserviços
AWS comoAmazon EC2, Amazon RDS, e Elastic Load Balancing
•Remove o requisitode lançarmanualmenteosrecursosAWS necessários
pararodarsuaaplicação
•Vocêfazupload da suaaplicaçãoenquantoo Elastic Beanstalk cuidados
detalhesde provisionarcapacidade, balancamentode carga, escalabilidade
e monitoramentoda saúdeda aplicação
•Suportea aplicaçõesDockermulti-container (emconjuntocom Amazon
EC2 Container Service)

19. AWS Elastic Beanstalk
App
ELB
AZ
your-app.elasticbeanstalk.com
Alert
Log
Mon

20. AWS Elastic Beanstalk
App.zip
Python 3
WSGI entrypoint:
app.py
Python
libs
Façadeploy do seucontainer de trêsformas

21. AWS Elastic Beanstalk
App.zip
Python 3
WSGI entrypoint:
app.py
Python
libs
Dockerfile
•Imagemseráconstruída
emcadainstância

22. AWS Elastic Beanstalk
App.zip
Python 3
WSGI entrypoint:
app.py
Python
libs
Dockerrun.aws.json
•Manifesto quedescreve
comorodaro container

23. AWS Elastic Beanstalk
App.zip
Python 3
WSGI entrypoint:
app.py
Python
libs
Dockerrun.aws.json

24. AWS Elastic Beanstalk
Zip com contextoda aplicação
App.zip
-------------------------------
|--Dockerfile
|--Dockerrun.aws.json
Dockerfile
Dockerrun.aws.json

25. OpsWorks
•Criadoparafuncionarcom o OpsCodeChef, quepermitedeploy e
gerenciamentode aplicaçõesdiversas
•Permitea definiçãode stacksparaconfigurara infraestruturaem
AWS usando“receitas” Chef
•Uma “stack” podeconterum conjuntode recursosqueservema um
propósitoúnico, comoumaaplicaçãoweb
•As “receitas” Chef utilizadasno AWS OpsWorkspodeminstalaro
Dockernasinstânciase referenciarum Dockerfileparaconfiguraro
container apropriadamente

26. Adicionarlayer customizada
•Criadoparafuncionarcom o OpsCodeChef, quepermitedeploy e
gerenciamentode aplicaçõesdiversas
•Permitea definiçãode stacksparaconfigurara infraestruturaem
AWS usando“receitas” Chef
•Uma “stack” podeconterum conjuntode recursosqueservema um
propósitoúnico, comoumaaplicaçãoweb
•As “receitas” Chef utilizadasno AWS OpsWorkspodeminstalaro
Dockernasinstânciase referenciarum Dockerfileparaconfiguraro
container apropriadamente

27. Aplicarreceitaschef paraDocker

28. Pronto!

29. O desafiode operaremescala

30. Introduzindoo EC2 Container Service

31. EC2 Container Service
•Compatívelcom Docker
•Gerenciamentode Cluster facilitado
•Alta performance
•Eficiênciade usode recursos
•Extensível
•Seguro

32. ECS: Conceitos-chave
•Cluster
•AMIs “Container-Enabled”
•AgenteECS
•Gerenciadordo cluster (Cluster Manager)
•Agendadorde container (Container Scheduler)
•Definiçõesde Tarefas(Tasks/Task)

33. Task Definitions

34. Aplicaçõesdistribuídas
•Containers habilitamarquiteturasde micro-serviços
•Permitemapeamentode funçãoporcontainer
•Aderenteaosconceitosde baixoacoplamento(loose
coupling), elasticidade, escalabilidadee outros
princípiosde aplicaçõesdistribuídas
•Permitedeployments, provisionamentoe atualizações
rápidas

35. Customer Case –MercadoLibre

36. MercadoLibre in AWS

37. About MercadoLibre
•Operations in 13 countries.
•Leaders in the region.
•6.5 Billion MktCap(NASDAQ: MELI).
•2600 Employees (600 engineers).
•8
th
e-commerce site by traffic in the world.

38. About Me
•DaríoSimonassi
–Architecture Sr. Manager
–MercadoLibre.Com
–@ldsimonassi

39. About Me: Lately
•DaríoSimonassi
–Architecture Sr. Manager
–MercadoLibre.Com
–@ldsimonassi

40. MercadoLibre’stechnology stack
•MicroservicesArchitecture.
•REST API & Frontends.
•650 Applications.
•2 OnPremisedatacenters in the US.
•Private cloud: OpenStack15.000 VMs
•Fully self provisioned infrastructure services.

41. Frontends
REST API
MercadoLibre’sArchitecture
Items Users OrdersQuestions
Home Search
Item
Page
Checkout

42. MercadoLibre’sengineering teams
•Small teams.
•Full control & Ownership:
–Product decisions
–Development.
–QA (Automated).
–Deployment.
–Alerts & Production management.

43. Current technology stack
opportunities
•Great for:
–Empowerment.
–Flexibility.
•Opportunities:
–Complex.
–Lot of tools.
–Lot of knowledge required.
–Policies are difficult to enforce.

44. Fury
•Simplification
•Automation
•Unified platform
–Infrastructure provisioning
–Continuous integration.
–Configuration management.
–Dev& Tests environments.
–Real time metrics & logs.
–Alerting.

45. Fury Layout
Fury
Frontend
Fury CLI
AWS
Docker
External Services
(Metrics, APM, Logs,
GitHub, Alerting)

46. Operations
•Code
•Code, run, play & execute tests locally.
•Deploy
•CI, Fast & Safe code shipping.
•Monitoring & Alerting
•Know when something is going wrong.
•Understand what is going wrong.
•Notify the correct person.
•Follow up.
•Troubleshooting
•Take action to solve a problem.

47. Code
•Developers machine setup.
•Local setup for DB & Mock servers.
•Versions headache.
•Environment changes propagation.
•CI environment consistency.
•Developers don’t manage CI servers.
•What works in my machine must work in the CI.
•Environments fidelity.
•Use same libraries, same binaries.

48. Grails
SDK
Redis
Node.JS
API Mock
Browser
Fury CLI
$> fury run
Developer Machine
Text Editor
docker-compose.yml
Source Code

49. Code: Fury CLI Commands
•$> fury get homepage
•Will clone the homepage project repository.
•$> fury run
•Will run the webserver with all the dependencies locally.
•$> fury test
•Will run tests locally with all the dependencies.
•$> fury create-version 0.1
•Will push the code.
•Will tag the version.
•Will trigger CI and open browser window.

50. Deploy
•Risk of downtime
–Smooth transitions between versions.
–Real time metrics.
–Fast rollback.
•Reliability & Fidelity
–New Infrastructure vsUpdate scripts.
•Hardware usage efficiency
–Autoscaling.

51. Production Instance
•No Updates
•New application version -> New hardware
•No States
•No states like: on_duty, deploying.
•Instances are always supposed to be working.
•Single AMI
•Only one AMI that will run a container inside.
•The same container for the entire instance life.
•Minimal
•Only required stuff in this AMI.

52. Application DockerImage
Metrics
Gathering
Docker
Logs
Gathering
Docker
Productive Instance

53. ELB
Deployment: Before

54. ELB
Deployment: Starting

55. ELB
Deployment: In progress 0%

56. ELB
Deployment: In progress 6%

57. ELB
Deployment: In progress 12%

58. ELB
Deployment: In progress 25%

59. ELB
Deployment: In progress 50%

60. ELB
Deployment: In progress 100%
Fast Rollback Spare Pool

61. ELB
Deployment: Finishing

62. ELB
Deployment: Done

63. Monitoring & Alerting
•Alerting:
–To know fast and reliably when something is going wrong:
•No false alarms.
•Instance health no longer suitable.
•Notify the right persons & follow up.
•Diagnostics:
–To be able to understand what is going wrong.
•Metrics
–Multidimensional.
–Real-Time.
•Logs.

64. Alerting: Based on metrics
•HTTP:
–Errors.
–Response time.
•Infrastructure:
–How many machines stopped working unexpectedly during
the last hour?
•Business:
–Is the application doing what it should?
–Compare vsyesterday / last week.

65. Alerting: Based on metrics

66. Alerting: Based on metrics

67. Alerting: No false alarms
•Reasonable thresholds.
–Undesirable but expected situations shouldn’t be an alert.
–You won’t improve your application SLA on Saturday 3AM.
•Different thresholds for different periods.
–Alert on > 25% errors in the last minute.
–Alert on > 5% errors in the last hour.
–Alert on > 0.2% errors in the last day.

68. Alerting: Notifications.
•Teams manage the duty schedule.
•Escalation & Backup is necessary.
•Automatic Alerts follow up.
•Flexibility & Mobile.
–“I’m going to watch a movie, can someone get my
duties for the next three hours?”

69. Alerting: Notifications.

70. Diagnostics: Metrics
•Real Time.
–You’ll be changing things and you’ll need feedback.
•Multidimensional
–Tagged metrics will really make the difference when trying
to identify the failure.

71. Diagnostics: Metrics key features

72. Diagnostics: Logs
•Useful logs
–Events.
–Unusual errors.
–Avoid traces.
•Tagged
–Good tagging will allow you to make sense of the logged
data.

73. Logs: Tagging

74. Troubleshooting: Actions
•Deploy
–We’ve deployed a bug.
–We need to rollback fast or deploy a new version fast.
•Scale
–There is a performance issue, we need to increase the compute power.
–There is a specific bug, we need to block traffic.
•Restart
–There is a leak (memory, connections etc).
–Application restarting is a suitable yet temporary workaround for
those cases.

75. Fury: Key AWS Services
•Networking & Availability:
–SDN (Network Isolation/Segmentation)
–Multi AZ
–Direct Connect (OnPremiseDC Interop).
•Deploy:
–ELB, ASG, CloudWatch.
•Cache:
–ElasticCache, DynamoDB.
•Resources Orchestration:
–CloudFormation(For external resources too).

76. Fury: Technologies & Services
•Environment Management:
–Docker/ Dockercompose
–Jenkings.
•Diagnostics and Alerting:
–Datadog.
–Opsgenie.
–New Relic.
•Repositories:
–GitHub
–DockerHub.
•Logs Management:
–Elastic Search + Kibana

77. Referências
Docker:
http://www.docker.io/
http://pt.slideshare.net/dotCloud/docker-intro-november
OpsCodeChef:
https://www.chef.io/chef/
https://github.com/bflad/chef-docker
Amazon Web Services
http://aws.amazon.com/ecs/
http://aws.amazon.com/elasticbeanstalk/
http://aws.amazon.com/opsworks/
http://aws.amazon.com/ec2/
http://aws.amazon.com/cloudformation/

78. Perguntas?

79. Obrigado!