[IJCAI2023] A Scalable and Adaptive System to Infer the Industry Sectors of Companies: Prompt + Model Tuning of Generative Language Models

The Private Equity (PE) firms operate investment funds by acquiring and managing companies to achieve a high return upon selling. Many PE funds are thematic, meaning investment professionals aim to identify trends by covering as many industry sectors as possible, and picking promising companies within these sectors. So, inferring sectors for companies is critical to the success of thematic PE funds. In this work, we standardize the sector framework and discuss the typical challenges; we then introduce our sector inference system addressing these challenges. Specifically, our system is built on a medium-sized generative language model, finetuned with a prompt + model tuning procedure. The deployed model demonstrates a superior performance than the common baselines. The system has been serving many PE professionals for over a year, showing great scalability to data volume and adaptability to any change in sector framework and/or annotation.

2. A Scalable and Adaptive System to Infer the
Industry Sectors of Companies: Prompt +
Model Tuning of Generative Language Models
Lele Cao Vilhelm von Ehrenheim Astrid Berghult Cecilia Henje Richard Anselmo Stahl Joar Wandborg Sebastian Stan Armin Catovic Erik Ferm Hannes Ingelhag

3. Agenda
●EQT Motherbrain
●Sector Framework
●Method
●Training
●Experiments
●System
●Recap

4. EQT Motherbrain
●EQT: Private Equity Fund
○Buys, grows & sells companies
○Venture Capital, Buyout, Infrastructure, etc.

●Motherbrain: Data & Machine Learning Platform
○Support investment professionals
○Merging and enriching data sources

●Read more about us → https://motherbrain.ai

5. Sector Framework
Why?
●Identifying promising macro-trends
○E.g. renewable energy, circular economy
●Finding investments within these macro-trends

What?
●Customized: no one-size-fit-all
●Hierarchical: a tree structure
●Dynamic: prone to change
●Imbalanced: varying granularity
●Low-resource: few labels available

6. Method




Company
Name
Tags
Description




Sector(s)
Financial
Service
Capital
Markets
…
Gaming
Audio and
MusicTech
Model
●Task: Assign each company to the most relevant sector.
●Input Features originating from multiple sources:
○Company Name
○Company Tags
○Company Description

7. Method




Sector(s)




Company
Klarna Bank AB
Buy-now-pay-later, shopping
an online payment platform designed to facilitate
cashless payments.
Financial
Service
Capital
Markets
…
Gaming
Audio and
MusicTech
●Task: Assign each company to the most relevant sector.
●Input Features originating from multiple sources:
○Company Name
○Company Tags
○Company Description
Model

8. Method




Sector(s)
Financial
Service
Capital
Markets
…
Gaming
Audio and
MusicTech
●Task: Assign each company to the most relevant sector.
●Input Features originating from multiple sources:
○Company Name
○Company Tags
○Company Description
●Input Template
Model




Company
Klarna Bank AB, concerns buy-now-pay-later and
shopping, is an online payment platform designed to
facilitate cashless payments.

9. Method




Sector(s)
Financial
Service
Capital
Markets
…
Gaming
Audio and
MusicTech
●Task: Assign each company to the most relevant sector.
●Input Features originating from multiple sources:
○Company Name
○Company Tags
○Company Description
●Input Template
●Generative Completion
T5




Company
Klarna Bank AB, concerns buy-now-pay-later and
shopping, is an online payment platform designed to
facilitate cashless payments. Sector: __________

10. Training
●Add random soft prompt to represent the task
to be learned.
Sector
[[0.1, -0.2, …, 0.4],
[0.3, -0.1, …, -0.5],
…
[-0.2, 0.1, …, 0.6]]
Soft Prompt
[[0.5, -0.1, …, 0.2],
[0.1, -0.2, …, -0.1],
…
[-0.3, 0.4, …, 0.2]]
Token Embeddings
Tokenize & Embed
Klarna Bank AB, concerns buy-now-pay-later and shopping, is an
online payment platform designed to facilitate cashless payments.
Sector: __________
PLM: T5 - Large

11. Training
●Add random soft prompt to represent the task
to be learned.
●First t’ steps: only tune soft prompt part.
Sector
[[0.1, -0.2, …, 0.4],
[0.3, -0.1, …, -0.5],
…
[-0.2, 0.1, …, 0.6]]
Soft Prompt
[[0.5, -0.1, …, 0.2],
[0.1, -0.2, …, -0.1],
…
[-0.3, 0.4, …, 0.2]]
Token Embeddings
Tokenize & Embed
Klarna Bank AB, concerns buy-now-pay-later and shopping, is an
online payment platform designed to facilitate cashless payments.
Sector: __________
PLM: T5 - Large

12. Training
●Add random soft prompt to represent the task
to be learned.
●First t’ steps: only tune soft prompt part.
●Then, tune the entire architecture jointly until
convergence.
Sector
[[0.1, -0.2, …, 0.4],
[0.3, -0.1, …, -0.5],
…
[-0.2, 0.1, …, 0.6]]
Soft Prompt
[[0.5, -0.1, …, 0.2],
[0.1, -0.2, …, -0.1],
…
[-0.3, 0.4, …, 0.2]]
Token Embeddings
Tokenize & Embed
Klarna Bank AB, concerns buy-now-pay-later and shopping, is an
online payment platform designed to facilitate cashless payments.
Sector: __________
PLM: T5 - Large

13. Training
The annotations are heavily unbalanced.
fintech biotech energy food

14. Training
The annotations are heavily unbalanced.
●Augment classes with few labels using EDA
1
,
which randomly does:
○Synonym Replacement
○Random Insertion
○Random Deletion
○Random Swap
fintech biotech energy food
fintech biotech energy food
1: EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks
(Wei & Zou, EMNLP-IJCNLP 2019)

15. Training
Some sectors have extremely few annotations. To
maximize utilization of labels, we collapse these sectors
into their parents.
Keep the granularity if we have enough labels.

16. Experiments
Impact of Methods and Model Sizes
●Our method (Prompt + Model Tuning)
performs the best.
●All methods performs better with larger
model size.
●Our method (Prompt + Model Tuning) is
able to achieve better performance with a
smaller model size.
(a) Average Precision (a) Average Recall

17. Experiments
A snapshot of confusion matrix.
●Since L3 sectors are more fine-grained
requiring less (than L2) annotations, a
generally better performance is
observed for L3 than L2.
●Sectors on L3 levels have an accuracy
of over 90% except horizontal
software and vertical software.

18. System
1)Finetune on T5 PLM: when the sector framework is
changed or the annotation for any existing sector has
evolved significantly. Rerun inference for all.
2)Finetune on the latest trained model: when the sector
annotation only changed marginally.
3)Skip finetune: skip finetune and only run inference for
changed companies. Greatly reducing the daily inference
load (by approx 95%).

19. Recap
How did we tackle the challenges?
●Scarce and imbalance label: augmentation + label attribution
●Evolving labels and sector definitions:
○Leverage PLM with light-weight prompt tuning.
○Scalable and adaptive system design.
●Availability and quality of features: create input template
incorporating multiple sources of information.
●Make the system dynamic to changes in the taxonomy, labels or
data.

20. Thanks!
●For further questions and details, feel free to check out our paper:
https://arxiv.org/abs/2306.03313





●Or, email us: [email protected]


●Learn more about EQT Motherbrain at:
https://eqtgroup.com/motherbrain
https://eqtventures.com/motherbrain
https://motherbrain.ai/