We study the role of informal collaboration in academic knowledge production using published research papers previously presented and discussed at the NBER Summer Institute. We show that papers that have a discussant are published in highly-ranked journals and are more likely to be published in a to...
We study the role of informal collaboration in academic knowledge production using published research papers previously presented and discussed at the NBER Summer Institute. We show that papers that have a discussant are published in highly-ranked journals and are more likely to be published in a top journal. Conditional on having a discussant, the quality of a paper’s journal outlet increases in the discussant’s prolificness and editorial experience. This supports the idea that discussants help reduce information asymmetries that are inherent in the academic publication process. Conversely, using social network analysis we find no evidence that citations accumulate because discussants diffuse information about the paper.
Size: 812.88 KB
Language: en
Added: Aug 15, 2022
Slides: 52 pages
Slide Content
Discussants
Michael E. Rose
1
Daniel C. Opolot
2
Co-Pierre Georg
2
1
Max Planck Institute for Innovation and Competition, Munich, Germany
2
University of Cape Town, Cape Town, South Africa
1
Science is a social endeavor
Working with co-authors the norm in scienceWuchty et al. (Science 2007);
Chung et al. (QREF, 2009); Ductor (OBES, 2015)
There is alsoinformal collaborationLaband and Tollison (JPE, 2002)
Editors
Referees
Doctoral Advisers
Colleagues everywhere
Æ)Is informal collaboration part of the academic production function?
2
Science is a social endeavor
Working with co-authors the norm in scienceWuchty et al. (Science 2007);
Chung et al. (QREF, 2009); Ductor (OBES, 2015)
There is alsoinformal collaborationLaband and Tollison (JPE, 2002)
Editors
Referees
Doctoral Advisers
Colleagues everywhere
Æ)Is informal collaboration part of the academic production function?
2
Studies on academic collaboration and "paper" as a unit
formal informal
personAzoulay, Gra Zivin and
Wang (QJE, 2010)
Oettl (MS, 2012)
paperDuctor (OBES, 2015) Laband and Tollison (JPE,
2002)(aggregate counts only,
no causal attempt)
3
Laboratory: Discussants at NBER Summer Institutes
Annual, highly prestigious workshop series
Workshops organized by groups and programs
VSome workshops always havediscussants, the others never do
Focus on Finance for topical homogeneity
4
Q: Do Discussants matter?
1. Are papers with discussants at NBER SIs more success-
ful than those with general discussion?
2a. Are papers with more prolic discussants more success-
ful?
2b. Are higher citation counts of discussed papers due to
discussant's diusion eorts?
5
A: Discussants do matter
1. Are papers with discussants at NBER SIs more success-
ful than those with general discussion?
Yes
2a. Are papers with more prolic discussants more success-
ful?
Partly
2b. Are higher citation counts of discussed papers due to
discussant's diusion eorts?
No
6
Data
596 published papers presented at 85 workshops of NBER SIs of 12
Finance-related Groups between 2000 and 2009
Author and discussants characteristics: Euclidean index of citations,
Experience
Citation count(log) from Scopus, Top Journal Status, Journal Prestige
from SCImago
Neighborhood centrality in two collaboration networks
7
Are the groups comparable?
Same readability at presentation (= proxy for quality)
t-test
Same average aliation ranking of authors at presentation
t-test
Same average duration (55 min.)
t-test
Topically similar (based on cited journals)
9
Groups w/ and w/o discussants topically similarSimilarity = cosine of overlap of weighted counts of journals cited in
presented papers
weights obtained fromtdf-vectorization
details
All papers (in our sample): 0.09
All top 5 Econ papers: between 0.034 (AER) and 0.07 (Ectma)
EFCE ME IFM EFEL RISK PERE EFFE AMRE AP CF
EFCE 1
ME 0.95 1
IFM 0.8 0.8 1
EFEL 0.7 0.68 0.71 1
RISK 0.47 0.48 0.54 0.87 1
PERE 0.41 0.42 0.39 0.45 0.32 1
EFFE 0.39 0.44 0.42 0.47 0.4 0.21 1
AMRE 0.33 0.35 0.37 0.58 0.56 0.68 0.32 1
AP 0.32 0.31 0.42 0.82 0.88 0.24 0.44 0.57 1
CF 0.29 0.32 0.43 0.8 0.86 0.28 0.35 0.55 0.95 1
10
Groups w/ and w/o discussants topically similarSimilarity = cosine of overlap of weighted counts of journals cited in
presented papers
weights obtained fromtdf-vectorization
details
All papers (in our sample): 0.09
All top 5 Econ papers: between 0.034 (AER) and 0.07 (Ectma)
EFCE ME IFM EFEL RISK PERE EFFE AMRE AP CF
EFCE 1
ME 0.95 1
IFM 0.8 0.8 1
EFEL 0.7 0.68 0.71 1
RISK 0.47 0.48 0.54 0.87 1
PERE 0.41 0.42 0.39 0.45 0.32 1
EFFE 0.39 0.44 0.42 0.47 0.4 0.21 1
AMRE 0.33 0.35 0.37 0.58 0.56 0.68 0.32 1
AP 0.32 0.31 0.42 0.82 0.88 0.24 0.44 0.57 1
CF 0.29 0.32 0.43 0.8 0.86 0.28 0.35 0.55 0.95 1
10
Groups w/ and w/o discussants topically similarSimilarity = cosine of overlap of weighted counts of journals cited in
presented papers
weights obtained fromtdf-vectorization
details
All papers (in our sample): 0.09
All top 5 Econ papers: between 0.034 (AER) and 0.07 (Ectma)
EFCE ME IFM EFEL RISK PERE EFFE AMRE AP CF
EFCE 1
ME 0.95 1
IFM 0.8 0.8 1
EFEL 0.7 0.68 0.71 1
RISK 0.47 0.48 0.54 0.87 1
PERE 0.41 0.42 0.39 0.45 0.32 1
EFFE 0.39 0.44 0.42 0.47 0.4 0.21 1
AMRE 0.33 0.35 0.37 0.58 0.56 0.68 0.32 1
AP 0.32 0.31 0.42 0.82 0.88 0.24 0.44 0.57 1
CF 0.29 0.32 0.43 0.8 0.86 0.28 0.35 0.55 0.95 1
10
1. Are papers with discussants at NBER SIs more successful
than those with general discussion?
10
Interesting descriptives
592 observations (=published in a Journal in 2021 latest)
50% in top publications (JF, JFE, RFS, AER, JPE, QJE, RESTud, Ecmta)
60% with discussant; median experience = 11 years
Distribution
50% published between 2 and 5 years later
DistributionFull summary statistics
11
Identication assumptions
A1
A2
A3
sorting)
12
Empirical strategy
Top
i
Æ®0Å®1¢Paper
i
Å®2¢Authori,t¡1Å®3¢Group
i
ů¢DiscussioniŲi(1)
SJRi,tÆ®0Å®1¢Paper
i
Å®2¢Authori,t¡1Å®3¢Group
i
ů¢DiscussioniŲi(2)
log(1ÅCitationsi,2022)Æ®0Å®1¢Paper
i
Å®2¢Authori,t¡1Å®3Group
i
Å(3)
Å®4¢SJRi,tů¢DiscussioniŰtŲi
13
Discussants and Journal Status
Top (Finance) Top (Econ+Finance)
(1) (2) (3) (4)
Discussion 3 .399
¤¤¤
2.939
¤¤¤
1.330
¤¤¤
1.039
¤
(0.000) ( 0.000) ( 0.002) ( 0.075)
Constant ¡5.012
¤¤¤
¡5.264
¤¤¤
¡2.440
¤¤
¡1.393
(0.000) ( 0.000) ( 0.025) ( 0.126)
Paper controls X X X X
Author controls X X X X
Group control X X X X
JEL categories X X
N 593 580 593 593
Pseudo R
2
0.281 0 .464 0 .208 0 .275
AIC 515 .6 386 .2 674 .6 619 .5
Random inference 0 .0333
¤¤
0.0656
¤
0.1174 0 .1458
Note:Logistic regression. Standard errors clustered on NBER group or joint workshop.
14
Discussants and journal quality
SJR Avg. citations h-index
(1) (2) (3) (4) (5) (6)
Discussion 2 .182
¤¤¤
1.767
¤
0.968
¤¤¤
0.678
¤¤
33.00
¤¤
17.36
(0.005) ( 0.077) ( 0.002) ( 0.037) ( 0.010) (0.217)
Constant 0 .166 1 .523 0 .723 1 .159
¤
130.5
¤¤¤
156.7
¤¤¤
(0.897) ( 0.280) ( 0.186) ( 0.056) ( 0.000) (0.000)
Paper controls X X X X X X
Author controls X X X X X X
Group control X X X X X X
JEL categories X X X
N 576 576 576 576 576 576
R
2
0.277 0 .321 0 .261 0 .307 0 .162 0 .242
Random inference 0 .0949
¤
0.0656
¤
0.0910
¤
0.0783
¤
0.1321 0 .2133
Note:Logistic regression. Standard errors clustered on NBER group or joint workshop.
log transformation
15
Discussants and citation counts
log(1+Total citations)
(1) (2) (3) (4)
Discussion 0 .260
¤
0.108 ¡0.0294 ¡0.0146
(0.056) ( 0.392) ( 0.833) ( 0.889)
Constant 3 .367
¤¤¤
3.417
¤¤¤
3.410
¤¤¤
4.482
¤¤¤
(0.000) ( 0.000) ( 0.000) ( 0.000)
Paper controls X X X X
Author controls X X X X
Group control X X X X
Publication year FE X X X X
JEL categories X X X
Journal control X
Journal FE X
N 596 596 576 596
R
2
0.359 0 .386 0 .427 0 .584
Note:Logistic regression. Standard errors clustered on NBER group or joint workshop.
Negative Binomialsine hyperbolicusFlexible lags 16
Recap
Discussants help bring the paper to better journal on average
. . . but papers not better cited on average given the journal quality
)Papers get published where they belong
Next:
17
2b. Are higher citation counts of discussed papers due to
discussant's diusion eorts?
2b. Are higher citation counts of discussed papers due to
discussant's diusion eorts?
17
Identication assumptions
A4
A5
A6
Survey resultsAss. matching on seniorityAss. matching on characteristicsAss. matching on centrality
18
Discussants characteristics
Probe 350 papers with known discussants
Euclidean index of citations in year of discussion
Experience
Gender (via genderize.io)
Aliation rank
Practitioner (central bank, law rm, government)
Editorial experience (obtained from CVs)
in year of discussion
19
Editorial experience matters
...Æ®0Å®1¢Paper
i
Å®2¢Authori,t¡1Å®3Group
i
ů¢DiscussantiųgŰtŲi
Top journal: No eect of any characteristic
Journal quality:
Euclidean index (0.00240
¤¤¤
)
aliation rank (¡0.00279
¤¤
)
editorial experience (1.068
¤¤¤
)
Log Citation counts (w/ journal quality controls): No eect of any
characteristic
20
Editorial experience matters
...Æ®0Å®1¢Paper
i
Å®2¢Authori,t¡1Å®3Group
i
ů¢DiscussantiųgŰtŲi
Top journal: No eect of any characteristic
Journal quality:
Euclidean index (0.00240
¤¤¤
)
aliation rank (¡0.00279
¤¤
)
editorial experience (1.068
¤¤¤
)
Log Citation counts (w/ journal quality controls): No eect of any
characteristic
20
P(discussant cites "her" paper) < P(other participants cite it)0 5 10 15 20
Years until/since publication
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
Citation probability
Citation by
Same authors
Other workshop authors
Own discussant
Other workshop discussants
21
Neighborhood Centrality
What's the number of all neighbors of a researcheriwithin a given radius¿,
while discounting for distance with given information quality lter±?
Neighborhood
i(±)Æ
1X
¿Æ1
±
¿¡1
ki¿
!Measures ability to spread information
22
Neighborhood centrality in two networks
Formalnetwork linking authors on a paper in 370 Economics journals
a
Informalnetwork (CoFE) linking authors and acknowledged commenters on
papers from 6 Finance journals
b
Rose and Georg (RP 2021)
Papers from 2000-2011, network intuses publications fromt¡1,t,tÅ1
a
Journals ranked at least C by Combes and Linnemer (2010) in the following categories:
General Economics, C, D, E, F, G, H, K, and R
b
JF, RFS, JFE, JFI, JMCB, JBF
23
Neighborhood centrality in two networks
Formalnetwork linking authors on a paper in 370 Economics journals
a
Informalnetwork (CoFE) linking authors and acknowledged commenters on
papers from 6 Finance journals
b
Rose and Georg (RP 2021)
Papers from 2000-2011, network intuses publications fromt¡1,t,tÅ1
a
Journals ranked at least C by Combes and Linnemer (2010) in the following categories:
General Economics, C, D, E, F, G, H, K, and R
b
JF, RFS, JFE, JFI, JMCB, JBF
23
More central discussants6!higher citation count (informal net-
work)−0.20
−0.15
−0.10
−0.05
0.00
0.05 0.15 0.25 0.35 0.45 0.55 0.65 0.75 0.85 0.95
l
l
l
l l l l l l l
24
More central discussants6!higher citation count (coauthor net-
work)−0.4
−0.2
0.0
0.2
0.4
0.05 0.15 0.25 0.35 0.45 0.55 0.65 0.75 0.85 0.95
l
l
l
l
l l l l l l
25
Take-aways of "Discussants"
Informal collaboration widespread, poorly understood
Discussants = assigned informal collaborators
Having a discussant increases chances to publish high, though not citation
count
Discussants with editorial experience help the most, butnotbecause of
diusion eects
Thank you!
26
Appendix
26
Summary statistics of Journal sample
N Mean Median Std.Dev. Min Max
Total citations 596 170 .50 93.00 263.80 0.00 3418.00
Top publication 596 0 .51 1.00 0 .50 0.00 1 .00
SJR 576 8 .09 6.89 5 .14 0.16 22.54
# of pages 596 29 .62 29.50 11.56 5.00 87.00
# of authors 596 2 .18 2.00 0 .80 1.00 5 .00
Age 596 3 .52 3.00 2 .05¡1.00 14.00
Author total Euclid 596 431 .96 212.83 710.20 0.00 8311.96
Youngest author experience 596 6.89 5.00 6 .68 0.00 54.00
Oldest author experience 596 15.14 14.00 9 .76 0.00 54.00
Discussion 596 0 .63 1.00 0 .48 0.00 1 .00
p
74
2
Å18
2
Å40
2
Å74
2
Å180
2
Æ213.8
back
27
Summary statistics of Journal sample
N Mean Median Std.Dev. Min Max
Total citations 596 170 .50 93.00 263.80 0.00 3418.00
Top publication 596 0 .51 1.00 0 .50 0.00 1 .00
SJR 576 8 .09 6.89 5 .14 0.16 22.54
# of pages 596 29 .62 29.50 11.56 5.00 87.00
# of authors 596 2 .18 2.00 0 .80 1.00 5 .00
Age 596 3 .52 3.00 2 .05¡1.00 14.00
Author total Euclid 596 431 .96 212.83 710.20 0.00 8311.96
Youngest author experience 596 6.89 5.00 6 .68 0.00 54.00
Oldest author experience 596 15.14 14.00 9 .76 0.00 54.00
Discussion 596 0 .63 1.00 0 .48 0.00 1 .00
p
74
2
Å18
2
Å40
2
Å74
2
Å180
2
Æ213.8
back
27
Distribution of publication lag0 2 4 6 8 10 12 14
Years until publication
back
28
Similar readabilityWith Without
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
Gunning fog index
p = 0.87
Paper level
With Without
Gunning fog index
p = 0.96
Workshop averages
Readability linked to future citations; Economics Letters GF index = 16.08
Dowling, Hammami and Zreik (EL, 2018)
back 29
Similar average Tilburg aliation rankWith Without
0
20
40
60
80
100
Avg. affiliation rank
p = 0.64
Paper level
With Without
Avg. affiliation rank
p = 0.80
Workshop averages
Note:Rankings for nancial institutions not available.
back 30
Similar presentation durationWith Without
0
10
20
30
40
50
60
Duration (in min)
p = 0.41
Paper level
With Without
Duration (in min)
p = 0.32
Workshop averages
back
31
Distribution of discussant experience0 5 10 15 20 25 30 35
Discussant experience (in years)
0.0
0.2
0.4
0.6
0.8
1.0
Empirical cumulative density
0
1
2
3
4
5
6
7
8
Frequency of observation (in %)
back
32
tdf-vectorization to measure topical similarityPaper 1 cites journals A, B, C, D in entire period
Paper 2 cites journals C, D, E, F
Paper 3 cites journals C, F
0
B
B
B
B
B
B
B
B
B
@
1 0 0
1 0 0
1 1 1
1 1 0
0 1 0
0 1 1
1
C
C
C
C
C
C
C
C
C
A
tdf
¡¡¡¡¡¡¡¡!
vectorization
0
B
B
B
B
B
B
B
B
B
@
0.58 0 0
0.58 0 0
0.34 0.37 0.61
0.44 0.48 0
0 0.63 0
0 0.48 0.79
1
C
C
C
C
C
C
C
C
C
A
Cosine similarity between e.g. 1 and 2: 0.34
back
33
tdf-vectorization to measure topical similarityPaper 1 cites journals A, B, C, D in entire period
Paper 2 cites journals C, D, E, F
Paper 3 cites journals C, F
0
B
B
B
B
B
B
B
B
B
@
1 0 0
1 0 0
1 1 1
1 1 0
0 1 0
0 1 1
1
C
C
C
C
C
C
C
C
C
A
tdf
¡¡¡¡¡¡¡¡!
vectorization
0
B
B
B
B
B
B
B
B
B
@
0.58 0 0
0.58 0 0
0.34 0.37 0.61
0.44 0.48 0
0 0.63 0
0 0.48 0.79
1
C
C
C
C
C
C
C
C
C
A
Cosine similarity between e.g. 1 and 2: 0.34
back
33
tdf-vectorization to measure topical similarityPaper 1 cites journals A, B, C, D in entire period
Paper 2 cites journals C, D, E, F
Paper 3 cites journals C, F
0
B
B
B
B
B
B
B
B
B
@
1 0 0
1 0 0
1 1 1
1 1 0
0 1 0
0 1 1
1
C
C
C
C
C
C
C
C
C
A
tdf
¡¡¡¡¡¡¡¡!
vectorization
0
B
B
B
B
B
B
B
B
B
@
0.58 0 0
0.58 0 0
0.34 0.37 0.61
0.44 0.48 0
0 0.63 0
0 0.48 0.79
1
C
C
C
C
C
C
C
C
C
A
Cosine similarity between e.g. 1 and 2: 0.34
back
33
Discussants and journal quality, log transformation
log(SJR) log(Avg. citations) log(h-index)
(1) (2) (3) (4) (5) (6)
Discussion 0 .279
¤¤
0.198 0 .320
¤¤¤
0.202
¤
0.210
¤¤¤
0.102
(0.035) ( 0.177) ( 0.004) ( 0.068) ( 0.009) ( 0.321)
Constant 0 .899
¤¤¤
1.128
¤¤¤
0.256 0 .402
¤¤
4.848
¤¤¤
5.056
¤¤¤
(0.001) ( 0.000) ( 0.151) ( 0.049) ( 0.000) ( 0.000)
Paper controls X X X X X X
Author controls X X X X X X
Group control X X X X X X
JEL categories X X X
N 576 576 576 576 576 576
R
2
0.181 0 .266 0 .200 0 .264 0 .121 0 .209
Note:Standard errors clustered on NBER group or joint workshop.
back
34
Discussants and citation counts, Negative Binomial
(1) (2) (3)
Total citations
Discussion 0 .203 0 .0576 ¡0.0289
(0.121) ( 0.661) ( 0.849)
Constant 3 .829
¤¤¤
3.860
¤¤¤
3.854
¤¤¤
(0.000) ( 0.000) ( 0.000)
Paper controls X X X
Author controls X X X
Group control X X X
Publication year FE X X X
JEL categories X X
Journal control X
N 596 596 576
Pseudo R
2
0.0406 0 .0466 0 .0503
AIC 7033 .5 6989 .6 6756 .1
Note:Standard errors clustered on NBER group or joint workshop.
back 35
Discussants and citation counts, inverse hyperbolic sine
(1) (2) (3)
asinh(Total citations)
Discussion 0 .265
¤
0.110 ¡0.0349
(0.059) ( 0.393) ( 0.804)
Constant 4 .008
¤¤¤
4.052
¤¤¤
4.048
¤¤¤
(0.000) ( 0.000) ( 0.000)
Paper controls X X X
Author controls X X X
Group control X X X
Publication year FE X X X
JEL categories X X
Journal control X
N 596 596 576
R
2
0.356 0 .383 0 .426
Note:Standard errors clustered on NBER group or joint workshop.
back
36
Discussants and citation counts after varying durationYears since publication
−0.4
−0.2
0.0
0.2
0.4
0.6
base 0 1 2 3 4 5 6 7 8 9 10 11 12 13
l
l
l l l l
l
l l
l
l
l
l
l
l
Note:Standard errors clustered on NBER group or joint workshop.
back
37
Workshop organizers look for topical t and ability to presentExcerpts from 15 respondents responsible for 42 workshops:
Discussants should form "a basis for a lively, productive debate between
authors, discussants, and audience."
Discussants should not "too close to the author and if possible coming from
a dierent perspective"
They should have "No fear of authors (i.e., probably don't get a very junior
person to discuss a big shot, unless you know the junior person is fearless)."
There are "often authors of good papers that were not chosen for
presentation" which qualify as discussant
back
38
Do senior authors get senior discussants? 0 5 10 15 20 25 30
Authors' mean experience
0
5
10
15
20
25
30
35
40
Discussant experience
b = 0.13
= 0.11
0 10 20 30 40 50
Authors' max. experience
b = 0.14
= 0.096
AMRE
AP
CF
EFEL
IFM
PERE
RISK
back
39
No assortative matching with prolic discussants
Dis. experience Top-30 Univ. Dis. Euclid
neg. bin. logistic OLS
Author max. Euclid ¡0.00005 0 .001 ¡0.054
(0.730) ( 0.172) ( 0.216)
Author max. experience 0 .004 ¡0.023 ¡2.114
(0.590) ( 0.276) ( 0.330)
Author max. experience
2
0.00004 0 .0002 0 .067
¤¤¤
(0.588) ( 0.416) ( 0.004)
# of authors ¡0.031 0 .220 ¡1.173
(0.556) ( 0.148) ( 0.943)
Discussion year FE X X
NBER group FE X X
N 441 401 441
Adjusted R
2
0.206
Akaike Inf. Crit. 3,058.030 526.004
Note:Standard errors clustered on NBER group or joint workshop. Constant not reported.
back
40
No assortative matching with central discussants
Dis. neighborhood
coauthor informal
Author max. Euclid ¡0.004 ¡0.019
pÆ0.703 pÆ0.541
Author max. experience ¡0.282 ¡2.481
pÆ0.539 pÆ0.116
Author max. experience
2
0.006 0 .029
¤
pÆ0.194 pÆ0.087
# of authors 4 .141 15 .607
pÆ0.230 pÆ0.188
Discussion year FE X X
NBER group FE X X
N 441 441
Adjusted R
2
0.236 0.387
Note:Standard errors clustered on NBER group or joint workshop.
back
41