re:Invent 2023 - The Pragmatic Serverless Python Developer

Are you developing AWS Lambda functions with Python? Always looking for tools to make you more productive? What if you could hear directly from practitioners? This session covers an opinionated approach to Python project setup, testing, profiling, deployments, and operations. Learn about many open source tools, including Powertools for AWS Lambda—a toolkit that can help you implement serverless best practices and increase developer velocity. Join to discover tools and patterns for effective serverless development with Python. To maximize your learning experience, the session includes a sample application that implements what’s described.

Recording: https://www.youtube.com/watch?v=52W3Qyg242Y&t=9s

1. The pragmatic serverless Python developer Heitor Lessa OPN305-R1 (he/him) Chief Architect, Powertools for AWS AWS Ran Isenberg (he/him) Principal Software Architect CyberArk

2. Pragmatism Use case Product API Stream processor Wrap - up Readiness

3. Pragmatism: Analysis paralysis Infra as code framework decisions & trade-offs situational awareness is key Project structure Domain-driven design? Micro or mono functions? Open source tools? Sync or async? Asynchronous testing? App integrations Observability? Leading practices? Metrics? Alarms? Documentation?

4. Pragmatism: Open source tools mkdocs-material pytest Pydantic Powertools for AWS Lambda p re-commit YAPF Tuna Radon Xenon AWS CDK mkdocstrings isort flake8 poetry cdk-monitoring-constructs codecov actionlint gitleaks GitPython typing-extensions ` mypy-boto3 cdk-nag cachetools aws-lambda-env-modeler mypy pytest-socket markdownlint dependabot Semantic PR

5. Pragmatism: Leading practices (CI) Project setup pre-commit check Merge conflicts Valid Python code JSON/YAML/TOML Secrets linting Code linting pre-pull request +pre-commit checks Test (unit/contract) AppSec baseline Complexity baseline Docs linting Pull request checks Governance checks Test (integ/E2E) SAST +pre-PR checks Dependency CVE code checkout virtual env dependencies pre-commit hooks

6. Pragmatism Use case Product API Stream processor Wrap - up Readiness

7. Use case: Simple yet practical Product API Product database Product events

8. Use case: Product API Product database Product events Route handlers CREATE Product LIST Products GET Product DELETE Product Amazon API Gateway

9. Use case: Product database Route handlers CREATE Product LIST Products GET Product DELETE Product Amazon DynamoDB Amazon API Gateway Product events

10. Use case: Product events AWS Lambda stream poller AWS Lambda Stream processor Amazon EventBridge Route handlers CREATE Product LIST Products GET Product DELETE Product Amazon DynamoDB Amazon API Gateway

11. Use case: Synchronous path AWS Lambda stream poller AWS Lambda Stream processor Amazon EventBridge Route handlers CREATE Product LIST Products GET Product DELETE Product Amazon DynamoDB Amazon API Gateway Synchronous

12. Use case: Asynchronous path AWS Lambda stream poller AWS Lambda Stream processor Amazon EventBridge Route handlers CREATE Product LIST Products GET Product DELETE Product Amazon DynamoDB Amazon API Gateway Synchronous Asynchronous

13. Pragmatism Use case Product API Stream processor Wrap - up Readiness

14. Introduction Principal Software Architect AWS Serverless Hero Owner & Blogger @RanTheBuilder.Cloud

15. Product API: Practices Practices Project structure, infra as code Lambda handler practices Business domain Integration logic, AWS services, and more Unit, integration, end-to-end testing Amazon DynamoDB Route handlers GET /products GET /{product} PUT /{product} DELETE /{product} Amazon API Gateway

16. Synergy of elements DevOps culture DEV -> Production Faster development Project structure Project: Overview

17. Project repository structure reflects elements: Business domain: Lambda handlers code, business logic, and integrations code Tests: unit, integration, end to end Production readiness: spread across all folders pyproject.toml: one file to rule them all IaC: CDK, AWS SAM, Terraform, and more Project: Folder structure

18. We chose AWS CDK as IaC framework One CDK app and one stack Domain-driven approach: One construct for product REST API One constructs builds: API GW Lambda functions DynamoDB table All roles Project: Infrastructure as code

19. Start with one Lambda function All business domain code and logic What is the ”best” way to write a Lambda function? Amazon DynamoDB Route handlers PUT /{product} Amazon API Gateway Handler: Overview

20. Start with a handler that does it all Single file Gets input Handler: Event source parsing

21. Start with a handler that does it all Single file Gets input Domain logic/DB access Handler: Integration

22. Start with a handler that does it all Single file Gets input Domain logic/DB access Returns response Handler: Response

23. Start with a handler that does it all Single file Gets input Domain logic/DB access Returns response Context matters Cron job handler vs. REST API with logic Handler: One handler to rule them all?

24. As the service expands: Code duplication Testing becomes harder Code readability issues – 300+ lines This does not scale! We need a better way to develop complex services Amazon DynamoDB Route handlers GET /products GET /{product} PUT /{product} DELETE /{product} Amazon API Gateway Handler: Will my service evolve?

25. Business logic only May be shared by N handlers Isolated tests Handler: Architectural layers Handler No domain code Concise responsibilities Domain

26. Handler: Responsibilities Configuration Env. vars Validation Serialize output Call domain

27. access Amazon DynamoDB Amazon API Gateway Handler Domain invoke handle output serialize Handler: High-level event flow

28. Zero domain code Short and concise Clear responsibilities Handler: Event source parsing (after)

29. Zero domain code Short and concise Clear responsibilities Handler: Integration (after)

30. Zero domain code Short and concise Clear responsibilities Handler: Response (after)

31. Zero domain code Short and concise Clear responsibilities Handler: Refactored

32. Functional or OOP Interface: Create product Get product List products Delete product Domain: Product

33. Functional or OOP Interface: Create product Get product List products Delete product Access DynamoDB Return output Domain: Integration

34. Integration practice relates to: Code that accesses DynamoDB And in general: Code that accesses API Integration with AWS resources Integration with external APIs Amazon DynamoDB Route handlers GET /products GET /{product} PUT /{product} DELETE /{product} Amazon API Gateway Integration: Overview

35. Domain code contains integration code When can it backfire? Integration: Should domain integrate directly?

36. DB requirements change and evolve New queries are too slow What if we replace DynamoDB? Harder refactors No isolated testing Can we do better? Route handlers GET /products GET /{product} PUT /{product} DELETE /{product} Amazon API Gateway Integration: Testing and refactoring challenges Amazon Aurora Amazon DynamoDB

37. Adapter pattern Contains API/DB code Business logic only May be shared by N handlers Isolated tests Integration: A new architecture layer Handler No domain code Concise responsibilities Domain Integration

38. invoke handle serialize output Amazon DynamoDB Amazon API Gateway Handler Domain Integration: High-level overview Integration call access output

39. Integration: Agreeing on an interface Abstract interface Interface includes: Create product List products Get product Delete product

40. Integration: Implementation example Implements adapter pattern The only class that has DynamoDB code Isolated tests Aurora handler inherits interface

41. Integration: Refactoring domain Getter for concrete implementation Domain is not aware of the underlying DB Easy to switch to a different database Output conversion

42. DevX Never leave IDE Real cloud resources Local debug Amazon DynamoDB Route handlers GET /products GET /{product} PUT /{product} DELETE /{product} Amazon API Gateway Testing: Overview

43. No deployment Run Locally Testing: Pyramid

44. call Test runner (pytest) Isolated unit Assertion Test data verify prepare Testing: Unit test overview

45. Test schema validations (Pydantic) Test isolated integration classes No deployment required Fast & repeatable Testing: Model validation

46. Require s deployment No deployment Run l ocally & on AWS Run l ocally Testing: Integration tests

47. call Test runner (pytest) Lambda handler Assertion Test data verify prepare resources Testing: Integration test overview

48. Build repeatable and consistent tests Event details: Print real events in CloudWatch Logs Powertools for AWS Lambda test JSON events AWS service documentation Testing: Event generation

49. Test-driven development (TDD) Breakpoints in IDE Work against cloud resources Testing: Local code, cloud resources

50. Test-driven development (TDD) Breakpoints in IDE Work against cloud resources Assert side effects and response Testing: Asserting responses

51. Test-driven development (TDD) Breakpoints in IDE Work against cloud resources Assert side effects and response Testing: Asserting side effects

52. Test-driven development (TDD) Breakpoints in IDE Work against cloud resources Assert side effects and response Testing: Integration test complete

53. Simulate failures Mock DynamoDB exceptions Assert handler response Why not write as a unit test ? Testing: Injecting SDK failures

54. Require deployment Require deployment No deployment Run on AWS Run l ocally & on AWS Run l ocally Testing: End-to-end (E2E) tests

55. call Test runner (pytest) Product API Assertion Test data verify prepare Endpoint Lambda function invoke Testing: E2E test overview

56. Simulate real user usage Runs 100% on AWS Testing: Black box testing

57. Simulate real user usage Runs 100% on AWS Assert response Testing: Asserting product creation

58. Simulate bad input API security tests: Invalid authorization Invalid authentication Testing: Unhappy paths

59. Pragmatism Use case Product API Stream processor Wrap - up Readiness

60. Amazon DynamoDB Stream processor: Intro Amazon EventBridge AWS Lambda Stream processor Practices Project scaffolding, infra as code Lambda handler practices Business domain Integration logic, AWS services, and more Unit, integration, end-to-end testing

61. Practices Same structure as CRUD API Project: Folder structure

62. Amazon DynamoDB Amazon EventBridge AWS Lambda Stream processor Handler: Focus area

63. Practices Extend your handler with defaults Handler: Extend the signature

64. Practices Extend your handler with defaults Build domain input from event Handler: Prepare and validate domain input

65. Practices Extend your handler with defaults Build domain input from event Inject integrations Handler: Create and inject event emitter

66. Practices Dependency injection to the rescue Domain: Use dependency and issue receipt

67. Amazon EventBridge Integration: High-level overview Integration layer Domain

68. Amazon EventBridge Integration: High-level overview Event Handler Domain

69. Amazon EventBridge Integration: High-level overview Event Handler Domain Model to Event Standardize Event Inject metadata

70. Amazon EventBridge Integration: High-level overview Event Handler Domain Model to Event Event Provider Standardize Event Inject metadata send

71. Amazon EventBridge Integration: High-level overview Event Handler Domain Model to Event Event Provider Standardize Event Inject metadata send PutEvents

72. Amazon EventBridge Integration: High-level overview Event Handler Domain Model to Event Event Provider Standardize Event Inject metadata Build request Error Handling Issue Receipt send PutEvents

73. Integration: Model to standard Event

74. Integration: Model to standard Event Model  Event

75. Integration: Model to provider request Model  Event  EventBridge

76. Integration Domain Handler AWS Lambda Amazon EventBridge Testing: Isolated tests, thanks to layering Amazon DynamoDB

77. Testing: Quick recap on our handler

78. Integration Domain Amazon EventBridge Amazon DynamoDB Testing: Unit testing Lambda handler AWS Lambda Handler Event source Domain input EventHandler

79. Testing: Preventing side effects Practices Fail tests for unexpected connections

80. Testing: FakeEventHandler Practices Fail tests for unexpected connections Use in-memory Fakes

81. Testing: Unit testing benefits Practices Fail tests for unexpected connections Use in-memory Fakes Test in isolation, and as a whole unit

82. Amazon DynamoDB Integration Amazon EventBridge Domain Testing: Unit testing domain AWS Lambda Handler Event source Domain input EventHandler Business rule Use integ.

83. Testing: Unit testing domain logic Practices Fail tests for unexpected connections Use in-memory Fakes Test in isolation , and as a whole unit

84. Amazon DynamoDB Amazon EventBridge Integration Domain Testing: Unit testing integration AWS Lambda Handler Event source Domain input EventHandler Business rule Use integ.

85. Testing: Contract testing Practices Fail tests for unexpected connections Use in-memory Fakes Test model -> event -> provider request

86. Practices Fail tests for unexpected connections Use in-memory Fakes Test model  event  provider request Testing: Contract testing (event)

87. Testing: Contract testing (provider) Practices Fail tests for unexpected connections Use in-memory Fakes Test model  event  provider request

88. Testing: Asserting contract Practices Fail tests for unexpected connections Use in-memory Fakes Test model  event  provider request AWS SDK stubber for deeper validation

89. Amazon DynamoDB Amazon EventBridge Integration Domain Testing: Asserting asynchronous behavior AWS Lambda Handler Event source Domain input EventHandler Business rule Use integ. Init provider Build event Send event

90. Amazon EventBridge Testing: Listening to events AWS Lambda Amazon DynamoDB

91. Amazon EventBridge Testing: Creating new side effects AWS Lambda AWS Step Functions Amazon DynamoDB Amazon DynamoDB Intercept Event Testing Construct

92. Amazon EventBridge Testing: Querying intercepted events AWS Lambda AWS Step Functions Amazon DynamoDB Amazon DynamoDB Intercept Event Testing Construct pk sk data metadata receipt_id test_eventbridge_provider_send b30a7 ..# SAMPLE_NOTIFICATION #2023-10-08T20:17:02… {…} {…} b30a7 …

93. Testing: Using test name as event source Practices Fail tests for unexpected connections Use in-memory Fakes Test model  event  provider request Validate input with AWS SDK stubber Make it easier to trace test events

94. Testing: Asserting intercepted events Practices Fail tests for unexpected connections Use in-memory Fakes Test model  event  provider request Validate input with AWS SDK stubber Make it easier to trace test events Consider eventual consistency

95. Pragmatism Use case Product API Stream processor Wrap - up Readiness

96. Powertools for AWS Lambda: T oolkit Batch processing REST/GraphQL API Input/output validation Config management Secrets handling Idempotency Observability BYO middleware Self-documented schemas Feature flags Data extraction Caching best practices, for everyone Streaming *feature set may vary across languages Python | TypeScript | Java | .NET

97. Tuna: Visualizing import time (cold start)

98. Py-spy: Visualizing most freq. code path

99. Pyinstrument: Visualizing select code areas

100. Pragmatism Use case Product API Stream processor Wrap - up Readiness

101. Recap: Open source repository o pinionated reference and fully documented

102. Summary: Takeaways PROJECT HANDLER DOMAIN INTEGRATION TESTING Resources and template