CI CD in the age of machine learning by Sofia Calcagno

Sofia Calcagno presented CI/CD in the age of machine learning during a joint meetup between WiMLDS Paris and MLOps Paris on June 2024.

1. 1
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
CI/CD in the ML era
MLOps Paris x WiMLDS
05/04/2024

2. 2
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
Who am I?
Sofia Calcagno
Software and ML engineer

3. 3
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
CI/CD in the ML era



What deployment patterns for each
situation?
A topic extracted fom Culture MLOps

4. 4
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
Continuous *
DeployBuildTestCode
Continuous Integration
Continuous Deployment

5. 5
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
*non-exhaustive list
Why a CI/CD?
●Reduce human error.
●Help debugging through execution logs.
●Trust the process : make it repeatable, allow safe rollbacks.

To accelerate delivery

6. 6
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
A new artifact to build and deploy
Why is it different in an ML context?
Packaged code
Model
NEW

7. 7
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
A new event may justify the deployment of a new artefact
Why is it different in an ML context?
Code change Data change
NEW

8. 8
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
1.Where to train models?

2.Where to store model versions?

3.When to load the desired model version?

All this through three examples
My focus

9. 9
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
Design a lean ML CI/CD
Prompt: create an illustration for a presentation on MLOps. Imagine a complex system for deploying machine learning models (Co-pilot Designer)

10. 10
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
Making a prototype or
demonstrator
02

11. 11
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
I want to show marketing teams that ML can help them.

The use case: show that I can predict appeal for a product.


My challenges:
Go fast - Be frugal


Our needs

12. 12
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
Inside inference service
Where to train?
CI/CD pipeline
Code validation +
build artifact
Artifact to train
and predict
Data
Inference
InferenceModel
Inference service in
production

13. 13
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
In memory, during inference
Where can I store versions of the model?
Model is not persisted

14. 14
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
During inference, in-memory
When to load model version?
@app.get("/predict")
def predict(X):
model = Model()
X_train, y_train = get_train_data()
model.fit(X_train, y_train)
return model.predict(X)

Python code, training model before performing inference
1
2
3
4
5
6

15. 15
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
Sequence diagram to get a prediction

16. 16
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
Starting to develop a
product
03

17. 17
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
I want to build a product in an iterative way.

The use case: I want to serve my first models to my users.


My challenges:
Start to measure the product value.
Being able to pivot quickly based on first users feedbacks.
Our needs

18. 18
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
In the developer’s environment (like code)
Where to train?
CI/CD pipeline
Inference
artifact
Inference service
in production
Model
Inference
Data
Training

19. 19
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
Git
Where to store model versions?
*
* when your model < 100MB

20. 20
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
During deployment, with code artifact
When to load the desired model version?
FROM python:3.11-slim
COPY . /source
COPY model.joblib /model
WORKDIR /source
RUN pip install -r requirements.txt
CMD streamlit run main.py
Dockerfile copying model and code is the same
artifact
1
2
3
4
5
6

21. 21
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
Sequence diagram to get an inference

22. 22
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
Scaling
04

23. 23
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
I want to scale.

The use case : serving lots of users, I want to be able to test and update models
quickly.

My challenges :
Have a robust production - acting in case of drift - keep on deploying on demand.

Our needs

24. 24
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
In a production service
Where to train?
CI/CD pipeline
Inference
artifact
Inference
artifact
Model
Inference
DataTraining
artifact
Training
artifact
Training service
Inference
service

25. 25
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
Where to store model versions?

26. 26
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
Blob, S3, etc.
Where to store model versions?
Code of training service,
Saves produced model into a blob
1
2
3
4
5
6
7
8
9
10
11
12
def save_model(model, model_name):
with open("modle.pickle") as f:
pickle.dump(f , model)
container_client = ContainerClient(
account_url="https://mydatalake.blob.cor.windows.net/" ,
container_name="models",
credential="secret"
)
file_name = f"models/{model_name}.pickle"
blob_client = container_client.get_blob_client(file_name)
with open('model.pickle', 'rb') as f:
blob_client.upload_blob(f)

27. 27
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
During inference, if it changed
When to change model version?
@app.get("/predict")
def predict(X):
model_handler = ModelHandler()
model = model_handler.load()
return model.predict(X)

class ModelHandler:
def load(self):
if self.check_if_model_changed():
self.model_ = self._load()
return self.model_
Pseudocode of inference, that check if model was updated at each
inference to update it if necessary.
Alternative : load model at each inference, or at each start of the service
1
2
3
4
5
6
7
8
9
10
11

28. 28
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
Sequence diagram to get an inference

29. 29
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
Conclusion
05

30. 30
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
To summarize…
On the Data Scientist’s
computer
On a dedicated service
Where to train the model?
Git
On a dedicated storage
/ artifact repository
Where to store the model?
During deployment,
with the code artifact
When it changes
When to fetch the model?
In the inference API Not persisted In memory from disk

31. 31
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
๏CI/CD is a bit more complex with ML because there is a new artifact and a new source of change.

๏There is no just one way to perform CI/CD when in an ML context: choose your pattern depending on
your context and document then in Architecture Decision Records.

๏Software tools are not always enough to deploy ML - but they can be, at least temporarily.

๏A software with ML can look like all three examples during its lifecycle: CI/CD will need to change
with its needs.
Takeaways

32. 32
There is a better way
OCTO Part of Accenture © 2023 - All rights reserved
Thanks!


Download my book!
(in French)