Extensible Python: Robustness through Addition - PyCon 2024

A talk given at PyCon 2024 about how you can write sustainable Python by understanding dependencies, composability, open-closed principles, and extensibility. Also covers topics such as Event-Driven Programming and Plug-in based Architecture

1. Software is both archeology and time-travel

2. Your job is to deliver value in a timely manner

3. How do you enable the future to be just as efficient ?

4. Extensibility

5. ) Think about the humans

6. Extensibility Themes

7. 1) Reduce Commit Complexity

8. 2 ) Be Mindful of OCP

9. Open-Closed Principle

10. Code should be open to extension and closed for modification

11. When To Split Are easy things hard to do? Do you encounter pushback against similar features? Do you have consistently high estimates? Do commits contain large changesets? Are you mixing policy changes with mechanisms changes

12. Policies vs. Mechanisms

13. 3 ) Separate Policies and Mechanisms

14. From backoff documentation

15. F rom Flask documentation

16. 4 ) Data-Driven Designs

17. Data options Can you put your data in a collection? Configuration Files? Persistence Layers (i.e. databases, bucket storage, etc.)?

18. 5 ) Develop Libraries First

19. Good Library Design README-driven Tested Easily Consumable Clearly-stated opinions Releasable (w/ Documentation) Extensible Composable

20. Composability

21. From Flask documentation

22. 6 ) Create Building Blocks

23. Architectural Extensibility

24. Dependencies

25. Physical Dependencies

26. from typing import Callable @dataclass class Metadata: does_match: Callable [[ User ], bool ] if __name__ == "__main__" : run_self_test()

27. pip install requests

28. Physical Dependencies Hard-coded into source code Easy to follow from A to B Easily understandable Easy to find, hard to change Hard to substitute or mock out Understandable by static analysis tools

29. pipdeptree

30. pydeps

31. pyan3

32. What Happens If You Need Something To Change?

33. A B C

34. A B C

35. A B D

36. A B Whatever

37. Logical Dependencies

38. requests.post( "table-management/pizza-made" , { "id" : order, "pizza" : pizza.to_json() }) # meal is an abstract type meal: Meal = meal_factory.create( "pizza" )

39. Logical Dependencies Typically determined at run-time Readability suffers Debuggability suffers Hard to find, easy to change Easy to substitute or mock out Not understandable by static analysis tools Crucial for Abstraction

40. 7 ) Trade-off Dependencies Judiciously

41. Event-Driven Architectures

42. def complete_order(order: Order ): package_order(order) notify_customer_that_order_is_done(order) notify_restaurant_that_order_is_done(order)

43. Architectural Extensibility

44. Producer Transport Consumer

45. Producer Transport Consumer

46. pypubsub

47. from pubsub import pub def notify_customer_that_meal_is_done(order: Order ): # ... snip ... pub.subscribe(notify_customer_that_meal_is_done, "meal-done" )

48. from pubsub import pub def complete_order(order: Order ): package_order(order) pub.publish( "meal-done" , order)

49. 8 ) Use Event-Driven Architectures to Decouple Producers and Consumers

50. Pluggable Architectures

51. Plug-in examples pytest poetry hypothesis pylint

52. Plugging Into Algorithms

53. @dataclass class PizzaCreationFunctions: prepare_ingredients: Callable add_pre_bake_toppings: Callable add_post_bake_toppings: Callable

54. def create_pizza( pizza_creation_functions: PizzaCreationFunctions ): pizza_creation_functions.prepare_ingredients() roll_out_pizza_base() pizza_creation_functions.add_pre_bake_toppings() bake_pizza() pizza_creation_functions.add_post_bake_toppings()

55. pizza_creation_functions = PizzaCreationFunctions( prepare_ingredients=mix_zaatar, add_pre_bake_toppings=add_meat_and_halloumi, add_post_bake_toppings=drizzle_olive_oil ) create_pizza(pizza_creation_functions)

56. pizza_creation_functions = PizzaCreationFunctions( prepare_ingredients=cook_bulgogi, add_pre_bake_toppings=add_bulgogi_toppings, add_post_bake_toppings=garnish_with_scallions_and_sesame ) create_pizza(pizza_creation_functions)

57. Plugging Into Packages

58. Stevedore

59. from abc import abstractmethod from typing import runtime_checkable, Protocol @runtime_checkable class UltimateKitchenAssistantModule( Protocol ): ingredients: list [ Ingredient ] @abstractmethod def get_recipes() -> list [ Recipe ]: raise NotImplementedError @abstractmethod def prepare_dish(inventory: dict [ Ingredient , Amount ], recipe: Recipe ) -> Dish : raise NotImplementedError

60. class PastaModule( UltimateKitchenAssistantModule ): def __init__(self): self.ingredients = [ "Linguine" , # ... snip ... ] def get_recipes(self) -> list [ Recipe ]: # ... snip returning all possible recipes ... def prepare_dish(self, inventory: dict [ Ingredient , Amount ], recipe: Recipe ) -> Dish : # interact with Ultimate Kitchen Assistant

61. from stevedore import extension def get_all_recipes() -> list [ Recipe ]: mgr = extension.ExtensionManager( namespace= 'ultimate_kitchen_assistant.recipe_maker' , invoke_on_load= True , ) def get_recipes(extension): return extension.obj.get_recipes() return list(itertools.chain(mgr.map(get_recipes)))

62. from setuptools import setup setup( name= 'ultimate_kitchen_assistant' , version= '1.0' , # . ... snip .... entry_points={ 'ultimate_kitchen_assistant.recipe_maker' : [ 'pasta_maker = ultimate_kitchen_assistant.pasta_maker:PastaModule' ], }, )

63. 9 ) Use Pluggable Python for Extension Points

64. 0) Think about the humans 1) Reduce Commit Complexity 2) Be Mindful of OCP 3) Separate Policies and Mechanisms 4) Consider Data-Driven Design 5) Develop Libraries First 6) Create Building Blocks 7) Trade-off Dependencies Judiciously 8) Use Event-Driven Architectures to Decouple Producers/Consumers 9) Use Pluggable Python for Extension Points

65. 10 ) Don't Overextend Yourself

66. Who Am I? Principal Software Engineer Cloud Software Group Owner of Kudzera, LLC Author of Robust Python Organizer of HSV.py

67. Extensible Python: Robustness Through Addition Patrick Viafore