Multilevel Modeling Data analysis Models

Multilevel

1. Copyright © E.Y.Li 1 2021/11/4 Multilevel Data Analysis: Models and Applications Eldon Y. Li Chair Professor & Director Ph.D. Program in Business Chung Yuan Christian University, Taiwan http://www.calpoly.edu/~eli *** All right reserved. Reference to this document should be made as follows: Li, E.Y. “Multilevel Data Analysis: Models and Applications”, unpublished lecture, Chung Yuan Christian University, 2019 ***

2. Copyright (c) E.Y.Li 2 2021/11/4 Agenda About me Abstract Introduction to model analyses Why multilevel model? When multilevel model? How multilevel model? Application example 1 Application example 2

3. About Me Eldon Y. Li is a university chair professor and former department chair of MIS at National Chengchi University, an adjunct chair professor of Asia University in Taiwan, and a former professor and coordinator of the MIS program at College of Business, California Polytechnic State University, San Luis Obispo, California, USA. He was the dean of College of Informatics and the director of Graduate Institute of Social Informatics at Yuan Ze University in Taiwan, as well as professor and founding director at the Graduate Institute of Information Management at the National Chung Cheng University in Chia -Yi, Taiwan. He received his PhD from Texas Tech University in 1982. He is the editor-in-chief of several international journals. He has published more than 250 papers in various topics related to innovation and technology management, human factors in information technology (IT), strategic IT planning, software quality management, and information systems management. His papers appear in Journal of Management Information Systems, Research Policy, Communications of the ACM, Internet Research, Expert Systems with Applications, Computers & Education, Decision Support Systems, Information & Management, International Journal of Medical Informatics, Organization, among others. 2021/11/4 3

4. Copyright (c) E.Y.Li 4 2021/11/4 Conventional survey studies usually collect individual data from different groups and analyze them independently in each group, unless there is no significant group difference. In contrast, multilevel model analysis allows individual data with organizational differences to be included in one regression model by treating these differences as higher-level independent variables.  Such kind of model is known as hierarchical linear model (HLM). This lecture introduces various research models and discusses why, when, and how to perform multilevel model analysis. The applications of multilevel model analysis are elucidated using two published studies in information systems field. Multilevel Model Analysis: Method and Applications Abstract

5. Introduction – Model Analysis Relational model (predictive) Causal model (static) Behavioural model (dynamic) Process model (staging) Mediation model (partial vs. full) Moderation model Moderated mediation ( MoMe ) model Mediated moderation ( MeMo ) model Multilevel model Mixed model

6. Relational model (predictive) Source: Li, E.Y.*  and Soenen , L. (1994) " Dollar Value of the Yen and Stock Price Reactions in Japan ,"  Journal of Global Business   (U.S.A.), Vol. 5, No. 1, Spring, pp. 5-12. Topix Small Topix Large Topix Composite Nikkei 225 Yen/Dollar Exchange (-1) Yen/Dollar Exchange (-1.5) Yen/Dollar Exchange (-2) Yen/Dollar Exchange (-3)

7. Relational model (predictive) Source: Li, E.Y.* (1994) " Artificial Neural Networks and Their Business Applications ," Information & Management (Elsevier), Vol. 27, No. 5, October, pp. 303-313.

8. Causal model (static) Source: Wu, Y.L.,  Li, E.Y.* , and Chang, W.L. (2016) " Nurturing user creative performance in social media networks: an integration of habit of use with social capital and information exchange theories ,"  Internet Research  (Emerald), Vol. 26, No. 4, pp.869-900. ( SSCI )

9. Behavioral model (dynamic) Theory of Reasoned Action Source : Fishbein , M., &  Ajzen , I. (1975).  Belief, Attitude, Intention, and Behavior: An Introduction to Theory and Research . Reading, MA: Addison-Wesley. 

10. Process model (staging) Source: Bhattacherjee , A. and Premkumar , G. (2004). Understanding changes in belief and attitude toward information technology usage: a theoretical model and longitudinal test. MIS Quarterly , 28 (2), 229-254.

11. Mediation model B C A

12. Partial mediation model Source: Huang, Y.H.*, Li, E.Y. , and Chen, J.S. (2009.3) " Information Synergy As the Catalyst Between IT Capability and Innovativeness: Empirical Evidence from Financial Service Sector ," Information Research: An International Electronic Journal , Vol. 14, No. 1, pp. 1-11. Figure 1: Research Model

13. Full mediation model Source: Yen, H.J.R., Li, E.Y. and Niehoff , B. (2008.9). Do organizational citizenship behaviors lead to information system success? testing the mediation effects of integration climate and project management. Information & Management , 45 (6), 394-402.

14. Moderation model (conceptual) B C A D

15. Moderation model (statistical) B C A D D × A D × B C = β + β 1 A+ β 2 B+ β 3 D+ β 4 D × A+ β 5 D × B+ 

16. Moderation model Source: Ajzen , I. “TPB Diagram”, available at http://people.umass.edu/aizen/tpb.diag.html

17. Moderated mediation model Source: Edwards, J. R., & Lambert, L. S. (2007). Methods for integrating moderation and mediation: a general analytical framework using moderated path analysis. Psychological methods, 12(1), 1. . The MoMe model occurs when the treatment effect of an independent variable A on an outcome variable C via a mediator variable B differs depending on levels of a moderator variable D. Specifically, either the effect of A on the B, and/or the effect of B on C depends on the level of D. But, there is no overall moderation of A on C. (Wikipedia, 2019) D A C B D

18. Moderated mediation model Source: Yen, H.J.R., Thi , H.P., and  Li, E.Y.*  (2021).  Understanding customer-centric socialization in tourism services .  Service Business  (Springer), accepted for publication on 13 October 2021. ( SSCI )

19. Mediated moderation model Source: Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations.  Journal of personality and social psychology ,  51 (6), 1173. The MeMo model occurs when the treatment effect of an independent variable A on an outcome variable C via a mediator variable B differs depending on levels of a moderator variable D. Specifically, either the effect of A on the B, and/or the effect of B on C depends on the level of D. And, there is an overall moderation of A on C. D A C B

20. Mediated moderation model Source: Venkatesh , V., Morris, M.G., Davis, G.B., and Davis, F.D. (2003). User Acceptance of information technology: Toward a unified view. MIS Quarterly , (27:3), 425-478.

21. Mediated moderation model Source: https:// arxiv.org /pdf/1701.08862 .

22. Mediated moderation model (generalized) Source: https:// arxiv.org /pdf/1701.08862.

23. Mixed model Source: DeLone , W. and McLean, E. (2003). The DeLone and McLean Model of information systems success: a ten-year update. Journal of Management Information Systems , 19 (4), 9-30.

24. Mixed model Source: Wixom, B.H. and Todd, P.A. (2005). A theoretical integration of user satisfaction and technology acceptance. Information Systems Research , 16 (1), 85-102.

25. Multilevel model Source: Li, E.Y.* and Shani, A.B. (1991 Spring) "  Stress Dynamics of Information Systems Managers: A Contingency Approach ,"  Journal of Management Information Systems   (ME Sharpe), 7 (4), 107- 130.

26. Multilevel model Source: Li, E.Y.* , and Ko, S.-F.(2021). Employee's market orientation behavior and firm's internal marketing mechanism : A multilevel perspective of job performance theory . Sustainability (MDPI), 13(12), 6972, 1-25.

27. Why multilevel model? Company 1 Company 2 Company 3 Company 30 Company 4

28. When multilevel model? Condition for applying multilevel analysis Variance between companies  relatively large Variance within a company (between employees)  relatively small. If variance between companies is small, merge data and use single level analysis. If variance within a company (between employees) is large, remove outliers or remove this company from the data.

29. When? (Cont.) Intraclass agreement index ( r wg ) Also called Within-organization agreement index r wg (J) value indicates the degree to which the responses to a measurement scale by members of the same organization converge. r wg (J) value >0.70 (James et al ., 1984) James LR, Demaree RG, Wolf G (1984) Estimating within-group interrater reliability with and without bias. J Appl Psychol 69:85 response

30. When? (Cont.) Intraclass correlation coefficients ICC1 compares the between-organizations variance with the within-organization variance to indicate the portion of variance in individual responses (MSW) accounted for by the between-organizations difference (MSB). > 0.12 ( Bliese 2000) K = the average sample size from a company Bliese , P.D. Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K.J. Klein and S.W.J. Kozlowski (eds.), Multilevel Theory, Research, and Methods in Organizations . San Francisco: Jossey -Bass, 2000, pp. 349–381.

31. When? (Cont.) Intraclass correlation coefficients (cont.) ICC2 reveals the reliability of the mean of an organization-level variable . If low reliability, multilevel analysis is not needed. That is, MSB should be large, MSW should be smaller and no more than 40% of MSB. > 0.60 ( Bliese 2000)

32. How multilevel model? Source: Huang, M.H., Li, E.Y. *, and Wong, C.S. (2015) " A Multilevel Model of Information System Success in the User Department: Integrating Job Performance Theory and Field Theory ," International Journal of Information Systems and Change Management ( Inderscience ), Vol. 7, No. 4, pp. 286-307.

33. How? (Cont.)

34. Copyright © E.Y.Li 34 2021/11/4 Application example 1 " A Multilevel Model of Information System Success in the User Department: Integrating Job Performance Theory and Field Theory "

35. Application example 1 - Originality The IS Success (ISS) model of DeLone and McLean (1992, 2003) IS’s Qualities User’s Satisfaction Organization’s Net Benefits IS Developers Users ??? 2021/11/4 Copyright © E.Y.Li 35 User Managers

36. Research questions What are the factors influencing net benfits of user department’s IS appcliation (evaluated as user manager’s job performance)? What are the interaction effects of these factors on net benefits?

37. Originality (Cont.) The ISS model of DeLone and McLean (1992, 2003) prescribes IS’s quality (including information, system, and service qualities), user’s satisfaction , and organization’s net benefits as the three integrated components. While IS developer is responsible for IS quality, users are concerned with satisfaction, leaving net benefits unattended. This study proposes user department’s IS performance be the surrogate of organization’s net benefits.

38. Underlying theories 2021/11/4 Copyright © E.Y.Li 38

39. Method - measures Job performance  User department’s IS performance. Opportunity  Top management support. Capability  User manager’s knowledge about IS applications. Willingness  User manager’s attitude toward IS applications. Legend:  =measured by.

40. Copyright © E.Y.Li 40 2021/11/4 Research model Source: Huang, M.H., Li, E.Y. *, and Wong, C.S. (2015) " A Multilevel Model of Information System Success in the User Department: Integrating Job Performance Theory and Field Theory ," International Journal of Information Systems and Change Management (Inderscience), Vol. 7, No. 4, pp. 286-307.

41. Solution model

42. Solution model UDISP ij represents the i th individual score of UDISP in j th organization. TMS j represents the aggregate score of TMS in j th organization. ISK ij represents the i th individual score of ISK in j th organization. ISA ij represents the i th individual score of ISA in j th organization. γ kl represents the slope of the k th level-1 predictor interacting with the l th level-2 predictor. U kj is a normal distribution and represents the residual of slope of k th level-1 predictor in the j th organization. r ij is a normal distribution and represents the residual of regression model in individual level.

43. Method - subjects Data requirements: Using similar information systems: ERP In similar industry: Manufacturing System experience: At least 1 year At least 10 companies in the industry: 42 At least 5 data ponits (departments) in each company: Average 6-7 deparments, Total 283

44. Analysis Focus group was used to ensure face validity of the survey questionnaire. Valid survey data were collected from 283 user managers and 42 top managers of 42 different Chinese manufacturing companies in which ERP systems were being utilized. Each company sample=6 to 7  K =6; ICC1 =0.253 >0.12; ICC2 =0.670 >0.60. The model can be validated by using Hierarchical Linear Modelling (HLM) software.

45. HLM software for Windows

46. Model specification in HLM Level-1 Level-2 Coefficients Predictors ---------------------- --------------- INTRCPT1, B0 INTRCPT2, G00 TMS, G01 ISK slope, B1 INTRCPT2, G10 TMS, G11 ISA slope, B2 INTRCPT2, G20 TMS, G21 Summary of the model specified (in equation format) --------------------------------------------------- Level-1 Model Y = B0 + B1*(ISK) + B2*(ISA) + R Level-2 Model B0 = G00 + G01*(TMS) + U0 B1 = G10 + G11*(TMS) + U1 B2 = G20 + G21*(TMS) + U2

47. HLM Outputs

48. Results Copyright © E.Y.Li 48 2021/11/4

49. Example 1 Conclusions Top management support, user-manager knowledge, and user-manager attitude all affect the level of UDISP significantly. (H1, H2, H3 supported) Top management support significantly moderates the relationship between user-manager attitude and UDISP. (H5 supported) The interaction effect of top management support and user-manager knowledge on UDISP is not significant. (H4 not supported)

50. Application example 2 A Multilevel Approach to Examine Employees' Loyal Use of ERP Systems in Organizations Copyright © E.Y.Li 50 2021/11/4

51. Application example 2 - Originality In the IS literature, organizational factors have been analyzed at the same level as individual factors. This study intends to breaks through the single-level lens. 2021/11/4 Copyright © E.Y.Li 51 Organizational Factors Individual Factors IS Success

52. Research questions What are the factors influencing employee’s loyal use of ERP system at the individual level and at the organizational level? What are the interaction effects of these factors on loyal use of ERP?

53. Underlying theories 2021/11/4 Copyright © E.Y.Li 53 Individual Level Organization Level Interactionism Paradigm Situational Strength Theory Employee’s Perceived Workload Social Information Processing Theory Social Learning Theory Rational Choice Theory Cost–Benefit Analysis Loyal Use Individual Perceptions Causal relation Construct Theoretical foundation Bottom-up/Top-down process

54. Research model Source: Yen, H.J., Hu, P.J.H., Hsu, S.H.Y., and Li, E.Y. * (2015) " A Multilevel Approach to Examine Employees' Loyal Use of ERP Systems in Organizations ," Journal of Management Information Systems (T&F), Vol. 32, No. 4, pp. 144-178.

55. Solution model Department-level Model: Employee’s Loyal Use : ELU ij = β 0j + β 1j ( EPB ij ) +β 2j ( EPW ij ) +r ij , Company-level Model: β 0j = γ 00 + γ 01 ( OLSQ j ) + γ 02 ( OLIQ j ) + γ 03 ( OLSOCB j ) +U 0j β 1j = γ 10 + γ 11 ( OLSQ j ) + γ 12 ( OLIQ j ) + γ 13 ( OLSOCB j ) +U 1j β 2j = γ 20 + γ 21 ( OLSQ j ) + γ 22 ( OLIQ j ) + γ 23 ( OLSOCB j ) + U 2j The final formula is as follows: ELU ij = γ 00 + γ 01 ( OLSQ j ) + γ 02 ( OLIQ j ) + γ 03 ( OLSOCB j ) +U 0j + ( γ 10 + γ 11 ( OLSQ j ) + γ 12 ( OLIQ j ) + γ 13 ( OLSOCB j ) +U 1j )  EPB ij + ( γ 20 + γ 21 ( OLSQ j ) + γ 22 ( OLIQ j ) + γ 23 ( OLSOCB j ) + U 2j )  EPW ij +r ij ,

56. Solution model ELU ij represents the i th individual score of ELU in j th organization. OLSQ j represents the aggregate score of OLSQ in j th organization. OLIQ j represents the aggregate score of OLIQ in j th organization. OLSOCB j represents the aggregate score of OLSOCB in j th organization. EPB ij represents the i th individual score of EPB in j th organization. EPW ij represents the i th individual score of EPW in j th organization. γ kl represents the slope of the k th level-1 predictor interacting with the l th level-2 predictor. U kj is a normal distribution and represents the residual of slope of k th level-1 predictor in the j th organization. r ij is a normal distribution and represents the residual of regression model in individual level.

57. Method - measures Loyal use  Proactive, extended use and willingness to recommend such uses to others . Benefits Workload System quality Information quality Service-oriented OCB

58. Method - subjects Data requirements: Using similar information systems: ERP Employee 39 scales; IS staff 13 scales. 15 employees and 5 IS staffs per firm Final sample: 485 employees, and 166 IS staffs System experience: At least 0.5 year At least 10 companies in the industry: 47 At least 5 data ponits (employees) in each company: Average 10-11 employees, Total 485

59. Analysis Focus group was used to ensure face validity of the survey questionnaire. Valid survey data were collected from 47 different Taiwanese companies in which ERP systems were being utilized. Each company sample=10 to 11  K =10; r wg IQ=.94 ; r wg SQ=.86 ; r wg SOCB=.94 >0.7 ICC1 IQ =0.14 ; ICC1 SQ =0.16 ; ICC1 SOCB =0.28 >0.12; ICC2 IQ =0.63 ; ICC2 SQ =0.66 ; ICC2 SOCB =0.58 >0.60.

60. Results Individual Level OL-IQ OL-SOCB Employee’s Perceived Workload Employee ’ s Loyal Use Organization Level OL-SQ H1 -0.11* ns -0.12* ns H2 ns 0.32*** ns ns 0.47*** Employee’s Perceived Benefits IL-IQ 0.11* -0.29*** IL-SQ IL-SOCB IL-SQ: 0.15*** IL-IQ: 0.13*** IL-SOCB: -0.02 0.14*   -0.07*** 0.23***

61. Results

62. Results

63. Results

64. Example 2 Conclusions IL-SQ, IL-IQ, and IL-SOCB have positive effects on PB, while IL-SQ has negative effect on PW. IL-SQ and IL-IQ have positive effects on ELU. PB has positive effect on ELU, while PW has negative effect. OL-SQ and OL-SOCB negatively moderate PB's positive effect on ELU. OL-IQ positively moderates PW's negative effect on ELU.

65. Overall Conclusions Multilevel model analysis overcomes the group differences and analyzes samples from multiple groups in one regression model . When group differences are significant, use multilevel level analysis; when not significant, use single level analysis. Test r wg , ICC 1 , ICC 2 before multilevel analysis. Remove outliers in each group before any analysis. Careful interpretation of results needs relevant experience.

66. Copyright (c) E.Y.Li 66 2021/11/4 Huang, M.H., Li, E.Y.*, and Wong, C.S. (2015) "A Multilevel Model of Information System Success in the User Department: Integrating Job Performance Theory and Field Theory," International Journal of Information Systems and Change Management (Inderscience), Vol. 7, No. 4, pp. 286-307. (EI) Yen, H.J., Hu, P.J.H., Hsu, S.H.Y., and Li, E.Y.* (2015) "A Multilevel Approach to Examine Employees' Loyal Use of ERP Systems in Organizations," Journal of Management Information Systems (T&F), Vol. 32, No. 4, pp. 144-178. (SSCI; FT50 Journal List) Extra readings

67. Copyright (c) E.Y.Li 67 2021/11/4 Q & A Thank You for Listening