Engineering @ Ava Women - Post-mortem story

This is overview of how Engineering worked in once potential unicorn startup Ava Women. Most of the slides are simply copy-paste from various presentations created by Ava’s engineering team during 2019-2022.

1. Advancing women‘s health 1
Software Engineering
@ Ava Women
Post-mortem story
Dejan Vukmirovic, 2024

2. Advancing women‘s health
Note:
Most of the slides and diagrams in this presentation
are simply copy-paste from various presentations
created by Ava’s engineering team during 2019-2022

3. Advancing women‘s health
CONTEXT

4. Advancing women‘s health 4
STARTUP LIFECYCLE
•Ava Women was the Swiss healthcare startup, founded in 2014.
•Created to improve women healthcare by accurately understanding their menstrual cycles.
•In 2017 and 2018 selected as winner in the Top 100 Swiss Startup Of The Year.
•In 2019 listed by the prestigious Tech Tour in 50 Europe’s most promising “super-scale ups”.
•… And then it was acquired in 2022 in bargain deal by FemTecHealth.

5. Advancing women‘s health 5
PRODUCT
Ava's machine learning algorithms interpret
physiological data in order to detect changes in
fertility status as well as screening for reproductive
health issues.
Perfusion
Breathing rate
Skin temperature
Heart rate variability
ratio
Resting pulse rate
Users wear a wearable
during sleep.
Intuitive mobile interface with
personalized and actionable
insights.
Ava Bracelet
or Wearable
Big Data & Artificial Intelligence Ava App5 Physiological Parameters
The bracelet measures 5
physiological parameters and collects
more than 1 million data points per
night.
Ava tracks over a million physiological parameters every night to provide
women with personalized, actionable insight into their health
Launching on Apple Watch in 2023

6. Advancing women‘s health 6
PORTFOLIO
HW2.0
HW2.1.x
Ava Fertililty
Tracker (TTC)
Ava COVID-RED
(CVD)
Ava Prevent
(CC)
HW1.2
FW TTC
FW TTC
FW CC
FW TTC
Original product, live since2017.
ForkofTTC, developedin 2020
duringCOVID19 pandemic.
More regulatedproduct,
in development2019-2022.

7. Advancing women‘s health 7
COVID-RED PRESENTED @ KEYNOTE SPEAK IN 2021 AWS re:Invent

8. Advancing women‘s health 8
ENGINEERING LANDSCAPE
•Located in Zurich (CHE) and Belgrade (SRB), with Customer Support in Makati (PHE).
•Peaked at 40+ people combined in Software Engineering, Data Science and Hardware.
snapshot of Org Tree from 2021

9. Advancing women‘s health
HEALTHCARE REGULATORY
COMPLIANCE

10. Advancing women‘s health 10
REGULATORY RULES
•Ava had to follow set of regulations to be available to customers:
•ISO 13485:2016 Medical Devices –Quality management systems –Requirement for regulatory purposes
•ISO 14971:2019 Medical Devices –Application of risk management to medical devices
•IEC 62304:2015 Medical device software –Software life-cycle processes
•EN ISO 10993 Biological evaluation of medical devices
•…
•In Europe:
•CE-marking approved
•Approved as Class IIaunder new EU Medical Device Regulation (MDR)
•MDSAP certified quality system (also accepted by FDA)
•In USA:
•First and only FDA-cleared fertility tracking wearable
•Successful 510(k) application

11. Advancing women‘s health 11
WORKING IN REGULATED DOMAIN
•All these regulations had impact on engineering:
•Establish a solid software development life cycle (SDLC).
•You need to prove that implementation follows the specification.
•You need to be able to trace user’s activities in specific time and with specific version of software.
•Ensure that data collected by the software during clinical trials and then in production
is accurate, secure, and protected against tampering.
•…
•Work with a lotof documentation, read input requirements and generate all kinds of documents (test
cases and plans, architecture decision records, source-code level documentation…)
•Attend a lot of meetings. Refinements, various planning sessions, retrospectives…

12. Advancing women‘s health 12
SDLC V-MODEL

13. Advancing women‘s health 13
IMPACT ON ENGINEERING
•We used the pressure of regulated environments to build “proper” engineering:
•Automate everything (possible).
•Do testing on multiple levels.
•Have proper input for work. Not JIRA tickets with only titles,
but with acceptance criteria, designs, user flows, links to architecture diagrams…
•Include engineers early in decision making about product.
•People would frequently complain to us in interviews that their current companies
don’t do engineering work properly, and our answer would be something like:
“Oh, we have it all. Complete user stories with acceptance criteria and designs, high-level requirements,
architecture diagrams, documented retrospectives, unit and integration tests… But you know what?
It comes with a price. You will have to attend a lot of meetings and do work not just on source code.
All of those don’t just create themselves out of nowhere.”

14. Advancing women‘s health
ARCHITECTURE

15. Advancing women‘s health 15
SYSTEM DECOMPOSITION(S)

16. Advancing women‘s health 16
TTC (AVA FERTILITY)

17. Advancing women‘s health 17
CC (AVA PREVENT)
publicly presented
internally used

18. Advancing women‘s health 18
DATA PROCESSING

19. Advancing women‘s health 19
AUDIT LOGGING
•Braceletlogs are stored as part of RAW file data
on AWS S3 for post processing.
•Audit logs come from mobile app or backend.
Stored in MongoDB for 6 months.
Moved permanently to AWS S3 afterwards.
•Logs and metrics are generated by backend.
•No PII data in any of logs.

20. Advancing women‘s health 20
TARGETED ARCHITECTURE
•Planned to be worked on
after the CC release (2022-…).
•To be a single platform
for running all Ava’s products.
•Also to support integration with
3rd-party wearables (Apple Watch…).

21. Advancing women‘s health
PROCESSES

22. Advancing women‘s health 22
STARTING POINT IN 2019

23. Advancing women‘s health 23
APPROACH
•Frequent changes in setups based on learnings and team retrospectives (go Agile!).
•Aligning process (battling) with:
•Regulatory restrictions/requirements
•External deadlines (marketing, hardware production…)
•Pivoting of company
•Manpower changes (downsizing)

24. Advancing women‘s health 24
PRODUCT DEVELOPMENT STAGES
RE
Architecture & Design
Implementation
Verification & Validation
Configuration Management
Project Management
Product Risk Management
Requirements Engineering
Scoping & Feasibility Definition of Design Input
Product Design
Dev&Verification
Design Output Validation
Project
Start
G I: Scope
Review
G II: Design Input G III: Design
Freeze
HW
Increments
SW
Increments
G IV: Product
Release
Mass
Production &
Launch
Sprint

25. Advancing women‘s health 25
Program
Flight Level 2
«End2End
Coordination»
Team
Flight Level 1
«Operational»
Story
Backlog
Task/Spike/Bug
Backlog
Domain
Team
Program Team /
Nexus Integration
Team
Portfolio
Flight Level 3
«Strategy»
Initiative/Epic
Backlog
Portfolio
Team
Portfolio Kanban
Apr May Jun Jul
Program Roadmap
Portfolio Roadmap
2019 2020 202120222023
Program Kanban
Team Kanban
S1S2S3S4S5
Sprints / Weekly Roadmap
Initiative
Epic (= all other)
Epic
Story
Story
Task / Spike / Bug
generate copy
Feature (= product
function)
Feature
User Story
User Story
FLIGHT LEVELS (FROM SAFE)

26. Advancing women‘s health 26
COMPLEX JIRA LOGISTICS
Filtering by
Project/Product
Filtering by
Team

27. Advancing women‘s health 27
2020-12 NOTES FROM ENGINEERING SESSION ON SETUP CHANGE
•Why Feature Teams did not work?
•Ambiguous User Stories that don’t fit Sprint cycle, and push for late integrations.
•Features not possible/likely to fit within each Team’s ownership (eg.microservice).
•Teams trying internally to separate their scope/code to match feature separation.
•Not possible to own end-to-end delivery (eg.release without waiting other team).
•Why Platform/Tech layout could work?
•Backend team could go Sprint (or so) in front of Mobile, with providing stable env that then can be used.
•Smaller groups with less distractions and clearer ownership.
•Concerns with Platform/Tech setup:
•More coordination needed on level above, between teams.
•How and with what group are planning sessions conducted?
•Lack of POs in company. This could completely lead to new setup failing.
•A big number of POs could lead to “politics” between platforms/teams.

28. Advancing women‘s health 28
FINAL SETUP
As per book:
Nexus as framework that is used for scaled agile projects.

29. Advancing women‘s health 29
FLOWS AND OWNERSHIP

30. Advancing women‘s health 30
PLANNING VS REALITY
Sprint 1 Sprint 2
Backend
Task 1
Backend
Task 2
Mobile
Task 1
Story done
Sprint 1 Sprint 2
Backend
Task 1
Backend
Task 2
Mobile
Task 1
Story done
Bug 1
DevOps
Task 1
Sprint N…
Bug 2

31. Advancing women‘s health 31
BURNDOWN CHART (6 MONTHS CYCLE)

32. Advancing women‘s health 32
CC –ESTIMATE/TIMELINE EVOLUTION

33. Advancing women‘s health 33
RETROSPECTIVES

34. Advancing women‘s health 34
“3 YEARS LONG” CC PROJECT RETROSPECTIVE SESSION
•2019: DevOps practices at it’s best
•2019: Great work in setting-up teams
•2020: Moved from DocumentDBto MongoDB
•2020: Technical stuff are well documented
•2020: PO re-joining the team
•2020: Casual December for recharge
•2020: AWS cost savings.
•2020: Yes we can! COVID-RED finished.
•2020: Terraform for HEW
•2021: New DS team
•2021: Good progress with automation tests.
•2021: Green/Red issue detected quickly
•…
•2019: Decision to go with new
over-engineered platform for CC
•2019: Not mocking Algo components
•2019: Feature teams
•2020: Stopped automation work
•2020: TTC work with poor documentation
•2020: People leaving
•2020: No Product Owner.
•2020: Parallel work on CC, CVD and TTC
•2021: Estimation of the timeline for CLV
•2021: Residual code still in Algo Worker
•2021: Lack of knowledge for documentation
•…

35. Advancing women‘s health 35
SCRUM MASTER’S ANALYSIS
•…evident problem is that we need to stop starting and start finishing
as is evident by this cumulative flow diagram.
•…
•We are filling in backlog at a faster rate than we are completing it
and there is a definitive Covid19 + layoffs impact as seen by reduced throughput.
•…
•Our task can take A LOT.
Half of them has a throughput time of a sprint, 70% of two sprints and 85% four sprints.
•Generally we tend to have a throughput of one task per day pretty stable,
meaning that we might need to rethink our planning.
•…
•(this is just a snippet)

36. Advancing women‘s health 36
MAPPING AND HANDLING TECH DEBT
Board for:
Android app on TTC platform

37. Advancing women‘s health 37
DOCUMENTATION FOR CC SUBMISSION
Stakeholder Needs 10
System Requirements 50
Use Cases / Requirements 400
Architecture Elements 100
Technical Specs / Bracelet Requirements700
Manual Test Cases 300
Automated Test Cases 3’000
Product Risks 230
Product Risk Controls 320
82
Documents that
Software & DS
teams deliver
5’081
Work items added
to Polarion
---------
---
---------
-----------

38. Advancing women‘s health 38SWFMEA
Product Risk
Control
Product Risk
11
PRODUCT RISK MITIGATION TRACES
S
t
r
i
c
t
l
y

c
o
n
f
i
d
e
n
t
i
a
l
Device RequirementsSpecification
Use Case
Requirement
(Basic Flow)
Requirement
(Alternative Flow)
Requirement
(NFRs, BR)
Requirement
(NFRs)
Subsystem Specification
Architectural
Element
Technical
Specification
Unit Specification
Architectural
Element
Technical
Specification
RISK MANAGEMENT (MATRIX AND TRACING IN DOCS)

39. Advancing women‘s health
TOOLCHAIN

40. Advancing women‘s health 40
STAGES AND AUTOMATION
Build
code
Static
code
analysis
Check
code
coverage
Doc
generate
Artifact
upload
Notify
Code review
Dev QA Integration Stage Production
Jenkins
Environments
Automated
testing
Build
ready
Formal
testing

41. Advancing women‘s health 41
TOOLS USAGE
Other Engineering Disciplines tools:
•Backlog Management (JIRA)
•Automated Testing (RestAssured, Gatling, Allure…)
•Release Management (Jenkins, Artifactory)
•Configuration Management (Git, Jenkins)
•Design (Zeplin, Invision)tc.)
ALM Engineering Disciplines covered (via Polarion):
•Requirements Engineering
•Architecture & Design
•Verification & Validation
•Product Risk Management
•Traceability Management
Required
for regulatory !

42. Advancing women‘s health 42
CONTROL OVER AWS COSTS

43. Advancing women‘s health 43
ELECTROMAGNETIC COMPATIBILITY (EMC) TESTING
To measures if Ava’s bracelets function satisfactorily in their electromagnetic environment
without introducing intolerable electromagnetic disturbance to anything in that environment.

44. Advancing women‘s health 44
POST-MARKET CLINICAL FOLLOW-UP (PMCF) STUDY
The aim of the study is to analyze:
1)if Ava users understand the meaning of the new LED colors
2)if the data recorded with the new hardware version (HW 2.1)
is equivalent to the current hardware version (HW 2.0).
Lead to notorious “PMCF bug” which:
1)took 3 months to resolve
2)and inspired developing of BLE Debug App

45. Advancing women‘s health 45
TESTING STRATEGIES
Test types
Functional tests Qualities tests Dynamic tests
Test levels
Acceptance testing
Acceptance criteria
in specification
Acceptance criteria
in specification
System testing
Manual -UI and API tests
Automation -UI and API tests
Exploratory tests
Load tests
(script, not automated)
Security tests (occasional PEN
by external company)
Integration testing
Manual -UI and API tests
Automation -UI and API tests
Exploratory tests
Test plan
Exploratory tests
Component testing
Code Review
Unit tests
On HW side:
By 3rd party company
that produces HW.
Manual -UI and API tests
Automation -UI and API tests

46. Advancing women‘s health 46
EXTERNAL VALIDATIONS FOR CC LAUNCH
Penetration testing
•Conducted by company “Alice and Bob”
•Targeted environment: Integration.
•Mix of scripted and manual testing.
•Result Findings:
•0 critical
•1 moderate
•8 low/informational
TLS / X.509 certificate chain
contains CA (root) certificate;
return only domain and intermediary
AWS well architected review
•Conducted by company “SwissCom”.
•No automation, it’s exploratory evaluation.
•Result Findings:
Pillar High riskMedium risk
Security 9 1
Reliability 2 5
Operational Excellence1 4
Performance Efficiency0 5
Cost Optimization 0 7

47. Advancing women‘s health 47
“PRCKO” COUNTER
•In-house developed “tool”.
•Counting user confirmed pregnancies in the TTC platform.
→ measuring success! Measuring user satisfaction!
•One in Zurich office
and one in Belgrade office.

48. Advancing women‘s health
TEAM SPIRIT

49. Advancing women‘s health 49
DIFFICULT POSITION
•In late 2018 overall engineering department had issues:
•Lack of trust between departments (Software, Data Science, Hardware, Support…).
•Lack of trust between offices.
•Individual concerns about company’s future.
•Individual feelings that “this company is no longer what it used to be”,
which was expected as company was fast growing.
•Incompatibility on individual levels as some people
were more suitable for early-phase-startup type of work
and not really for the expected new setup.
•And yet, team had to deliver on multiple streams
with deadlines and in more strict regulatory environment.

50. Advancing women‘s health 50
CAMARADERIE
•Ultimately… Very cohesive and motivated team was created.
•Team bonding was further strengthened due to pressure form external factors:
•COVID-19 lockdowns
•Financial turbulences in the company
•Downsizings that happened
•Merger with other companies, after purchase by FemtecHealth.

51. Advancing women‘s health 51
“NO MEETING” BLOCKS
With move to full-remote, due to COVID-19 pandemic, everything became a “meeting”.
This highly impacted team’s delivery, thus had to be handled.

52. Advancing women‘s health 52
RECRUITMENT PROCESS

53. Advancing women‘s health 53
RECRUITMENT STATS (2019)
Total days spent interviewing:
-1st round: 8 days

-2nd round: 10 days

54. Advancing women‘s health 54
2021 SURVEY “AVA AS EMPLOYER” –POSITIVE PERCEPTION
•Visible at the time of recruitment:
•We are product company. Not outsource.
•Meaningful domain. Not in gambling, betting, porn or similar.
•Engineering challenges are interesting.
•25 days vacation. Still more than most other companies are offering.
•Education budget. Rarely companies have such flexible option (“you choose how to spend”)
•(pre-COVID) Remote policy. 1/3 of days in a month without approval required.
•Positive impressions at the interview(s).
•Visible after joining:
•Big transparency on company activities and situations.
•Great bonding within the office.
•Experienced people to learn from.
•Good engineering practices in place (CI, code-review, documentation available…).

55. Advancing women‘s health 55
2020 SELF-EVALUATION: “THE JOEL TEST”
No. Question Score
1Do you use source control? 9
2Can you make a build in one step? 6
3Do you make daily builds? 0
4Do you have a bug database? 7
5Do you fix bugs before writing new code?1
6Do you have an up-to-date schedule? 9
7Do you have a spec? 8
8
Do programmers have quiet working
conditions?
3
9Do you use the best tools money can buy?0
10Do you have testers? 7
11
Do new candidates write code during their
interview?
9
12Do you do hallway usability testing? 4

56. Advancing women‘s health 56
2020 SELF-EVALUATION: “THE 12 FACTOR APP”
No. Question Score
1
One codebase tracked in revision control, many
deploys
8
2Explicitly declare and isolate dependencies6
3Store config in the environment 6
4Treat backing services as attached resources6
5Strictly separate build and run stages 8
6
Execute the app as one or more stateless
processes
4
7Export services via port binding 4
8Scale out via the process model 5
9
Maximize robustness with fast startup and
graceful shutdown
6
10
Keep development, staging, and production as
similar as possible
8
11Treat logs as event streams 8
12Run admin/mngtasks as one-off processes 0

57. Advancing women‘s health 57
2020 SELF-EVALUATION: “ATLASSIAN’S HEALTH MONITOR”
No. Question Score
1Full-time Project owner 0
2Balanced team 4
3Shared understanding 8
4Value and metrics 7
5Proof of concept 9
6One-pager 1
7Managed dependencies 6
8Velocity 9

58. Advancing women‘s health 58
YEARLY RETROSPECTIVES
•Introduced this concept to celebrate all the achievements
but also failures that happened during previous year.
•It also included metrics, some very useful and others just for fun.
•Each year entire team would vote in different categories
(most bizarre and nicest things happened, favorite vendor, our dream team…).
•Played a lot with Ava’s official graphical/branding resources.

59. Advancing women‘s health 59
YEARLY RETROSPECTIVES (SCREENS)

60. Advancing women‘s health
Gathering of Ava Belgrade alumni
after Engineering department shut-down