Orchestrating Data Transfers in the Digital Era: Navigating Challenges and Solutions for Multi-environment Data Pipelines

Check out these research slides to learn more about the extensive landscape of data movement across various environments.

Drawing insights from the new, comprehensive EMA report "Data in Motion: Orchestrating File Transfers and Data Pipelines in the Cloud Era," these slides highlight the critical role of data integration and management across diverse technological landscapes.

1. @ema _resea rch
| @ema_research
Orchestrating Data Transfers in
the Digital Era: Navigating
Challenges and Solutions for
Multi-environment Data Pipelines
Dan Twing
President and COO
Enterprise Management Associates (EMA)
Sponsored by

2. @ema _resea rch
| @ema_research 2
Watch the On-Demand Webinar
•Orchestrating Data Transfers in the Digital Era On-Demand Webinar:
https://info.enterprisemanagement.com/orchestrating-data-
transfers-in-the-digital-era-webinar-ws
•Check out upcoming webinars from EMA here:
https://www.enterprisemanagement.com/freeResearch
© 2024 Enterprise Management Associates, Inc.

3. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
Webinar Speaker
Dan joined EMA in 2005 and has over 25 years of
experience in information systems, software
development, and technology outsourcing. He focuses
on all aspects of intelligent and automated
management of IT.As President and Chief Operating
Officer of EMA, Dan is responsible for developing and
executing strategic market research, delivering value
to IT organizations through consulting engagements,
and directing product developments and marketing
efforts.
© 2024 Enterprise Management Associates, Inc. 3
Dan Twing
President and COO
EMA

4. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
Respondent Demographics
© 2024 Enterprise Management Associates, Inc. 4
1.5%
1.9%
3.4%
4.9%
5.3%
5.7%
7.6%
8.7%
9.1%
11.4%
40.2%
0.0% 10.0% 20.0% 30.0% 40.0% 50.0%
Infrastructure Engineer
Data Architect
Other Business Users
IT Cloud Platforms/Cloud Architect
Business Line Leader
Business Executive
IT Development
IT Security
IT Operations
IT Service Management (ITSM)
IT Executive
Role in Organization
Sample Size = 264
16%
30%
19%
18%
7%
10%
0%
5%
10%
15%
20%
25%
30%
35%
Number of Employees
NA
33%
Europe
31%
APAC
37%
1%
3%
5%
5%
7%
9%
9%
14%
45%
0% 10% 20% 30% 40% 50%
User of dashboards
Evaluate file transfer/data moving products
Technical user/operator of data moving products
Business IT analyst
Design, build, or operate in-house-developed file…
System designer
Recommend and specify related products/services
Financial decision-maker
Technical decision-maker
Role in selecting and implementing file transfer or data moving
technologies/solutions

5. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
File Transfer Needs Reach Across the Enterprise
© 2024 Enterprise Management Associates, Inc. 5
File transfers and data movement is a key part of
consistently delivering critical enterprise processes
80% IT Operations
76% Application Developers
67% Service
64% Sales
61% Supply Chain
59% Sales
58% Finance
56% Sourcing and Procurement
55% Asset Management
54% Country/Region specific
Degree functional areas rely upon
file transfers and data moving
(High or Very High):
95% IT Operations
91% Application Developers
85% R&D/Engineering
82% Service
82% Finance
80% Sales
80% Supply Chain
80% Sourcing and Procurement
79% Asset Management
74% Country/Region specific
Somewhat or Very proactive in
sponsoring new file transfers or
data moving initiatives:
Sample Size = 264

6. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research© 2024 Enterprise Management Associates, Inc. 6
Sample Size = 262
30%
37%
38%
39%
39%
42%
42%
46%
50%
0% 10% 20% 30% 40% 50% 60%
API-centric architecture
Increased emphasis on data quality
Adoption of advanced analytics and AI
Integrating/Supporting cloud services
Increased focus on data governance
Real-time data processing (streaming data)
Increased data volumes
Migration to the cloud
Increased focus on data security
How is digital transformation changing the requirements for moving data?
Digital Transformation Impact on Data Movement

7. @ema _resea rch
| @ema_research
Means of Data Movement
Timing, Environments,
and Methods

8. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
Timing of Data Movement
© 2024 Enterprise Management Associates, Inc. 8
48%
48%
57%
72%
78%
0% 20% 40% 60% 80% 100%
Ad hoc file sharing (on demand,
spontaneous)
Near-real time data integration
Event-driven data movement
Real-time data integration
Scheduled data movement
Data moving methods
Sample Size = 264
16.0%
16.9%
18.7%
23.4%
25.0%
0.0%5.0%10.0%15.0%20.0%25.0%30.0%
Ad hoc file sharing
Event-driven data movement
Near-real time data integration
Real-time data integration
Scheduled data movement
Distribution of data moved

9. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research© 2024 Enterprise Management Associates, Inc. 9
36%
38%
45%
48%
58%
63%
73%
0% 20% 40% 60% 80%
Third-party hosted
Mainframe
User devices
Distributed on-premises
Public cloud
SaaS
Private cloud
Environments from which data moves
Sample Size = 264
11.4%
12.5%
13.6%
14.5%
14.8%
16.1%
17.0%
0.0% 5.0% 10.0% 15.0% 20.0%
User devices
Third-party hosted
SaaS
Distributed on-premises
Mainframe
Public cloud
Private cloud
Distribution of data moved
Environments Moving Data

10. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
Software Used to Move Data
© 2024 Enterprise Management Associates, Inc. 10
33%
35%
37%
38%
42%
42%
43%
48%
48%
53%
58%
0%10%20%30%40%50%60%70%
Enterprise service bus (ESB) and integration
platforms
Middleware and message queue systems
In-house-developed scripts or applications
Managed file transfer (MFT) software
Integration platform as a service (iPaaS)
Workload automation (WLA)-native MFT
Automated workflow and BPA tools
API management platforms
Enterprise resource planning (ERP) systems
Secure file sharing and collaboration tools
File transfer protocol (FTP) servers
Types of data moving software in use
Sample Size = 264
6.8%
8.1%
8.4%
8.6%
9.1%
9.3%
9.4%
9.4%
10.1%
10.1%
10.7%
0.0%2.0%4.0%6.0%8.0%10.0%12.0%
Middleware and message queue systems
Enterprise service bus (ESB) and integration
platforms
Automated workflow and business process
automation tools
Enterprise resource planning (ERP) systems
API management platforms
In-house-developed scripts or applications
File transfer protocol servers
Secure file sharing and collaboration tools
Workload automation (WLA) software (using
native MFT features)
Integration platform as a service (iPaaS)
Managed file transfer (MFT) software
Distribution of data moved using each method

11. @ema _resea rch
| @ema_research
Data Movement
By File

12. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
Focusing Only On File-Based Data Movement Methods…
© 2024 Enterprise Management Associates, Inc. 12
15.8%
15.9%
15.9%
17.1%
17.1%
18.1%
14.5% 15.0% 15.5% 16.0% 16.5% 17.0% 17.5% 18.0% 18.5%
In-house-developed scripts or applications
Secure file sharing and collaboration tools
File transfer protocol servers
Workload automation (WLA) software (using native MFT features)
Integration platform as a service (iPaaS)
Managed file transfer (MFT) software
Distribution of Data Moved By File
Sample Size = 264

13. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
Most respondents have 1001-5000 file transfers sent or received weekly.
The weighted average is 14,308 file transfers weekly.
© 2024 Enterprise Management Associates, Inc. 13
0%
3%
3%
11%
19%
28%
13%
9%
5%
3%
5%
0% 5% 10% 15% 20% 25% 30%
Fewer than 10
10-50
51-100
101-500
501-1,000
1,001-5,000
5,001-10,000
10,001-20,000
20,001-50,000
50,001-100,000
More than 100,000
Weekly File Transfers
Sample Size = 264

14. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
File Transfers Increasing
© 2024 Enterprise Management Associates, Inc. 14
1%
5%
19%
34%
22%
17%
3%
0%
5%
10%
15%
20%
25%
30%
35%
40%
Increased
more than
75%
Increased
between
50% and
75%
Increased
between
25% and
50%
Increased
between
10% and
25%
Increased
less than
10%
Stayed the
same
Decreased
less than
10%
Decreased
between
10% and
25%
Decreased
between
25% and
50%
Decreased
between
50% and
75%
Decreased
more than
75%
Don’t know
How has the number of file transfers your organization sends/receives changed in the past year?
Sample Size = 264

15. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
57% are sending files to other countries
© 2024 Enterprise Management Associates, Inc. 15
No
42%
Yes
57%
Do you transfer files to other countries?
No Yes Don’t know
Sample Size = 264

16. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
Security is the most important feature when moving files (59%), followed by
Integration capabilities (49%) and High availability (47%)
© 2024 Enterprise Management Associates, Inc. 16
23%
32%
33%
36%
39%
40%
40%
42%
42%
43%
47%
49%
59%
0% 10% 20% 30% 40% 50% 60% 70%
Vendor support and training options
Detailed reporting and monitoring tools
Customization and extensibility of the solution
User-friendly interface for ease of use
Compliance and auditing capabilities (e.g., HIPAA, GDPR compliance)
Automation of file transfer processes and workflows
Cost-effectiveness and pricing flexibility
Support for multiple file types and sizes
Advanced encryption and data protection measures
Scalability to handle growth in data volume and connections
High availability and disaster recovery features
Integration capabilities with existing systems (e.g., ERP, CRM)
Security protocols (e.g., SSL/TLS, SFTP, FTPS)
Which of the following features do you consider most important when selecting a file transfer
solution for your organization?
Sample Size = 264

17. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
WLA Plays A Significant Role In File Transfers
© 2024 Enterprise Management Associates, Inc. 17
20%
20%
22%
22%
24%
0% 5% 10% 15% 20% 25% 30%
Not used in file transfer activities
WLA orchestrates FTP server software to transfer files
WLA orchestrates MFT software to transfer files
WLA orchestrates other data pipeline activities
Use native WLA MFT features to transfer files
Please indicate how WLA software is used in exchanging data or
transferring files.
Sample Size = 111

18. @ema _resea rch
| @ema_research
Securing File
Transfers

19. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research© 2024 Enterprise Management Associates, Inc. 19
Inbound data transfers
encrypted
Encrypted Unencrypted
47%
Outbound data
transfers encrypted
Encrypted Unencrypted
49%
Data at rest encrypted
Encrypted Unencrypted
41%
Sample Size = 264
Less Than Half Of File Transfers Are Encrypted

20. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research© 2024 Enterprise Management Associates, Inc. 20
Sample Size = 264
25%
25%
30%
31%
34%
35%
38%
50%
50%
61%
0% 10% 20% 30% 40% 50% 60% 70%
Hashing algorithms for data integrity verification
Data tokenization techniques
IPsec (Internet Protocol Security)
Encrypted tunnels (e.g., SSH tunnels)
AES (Advanced Encryption Standard) encryption
Public Key Infrastructure (PKI) for digital signatures and encryption
Transport Layer Security (TLS) protocols
Encrypted VPN
Secure Sockets Layer (SSL) protocols
Secure FTP or SFTP
Please select the encryption methods used during data movement between
systems, networks, or cloud environments.
Encryption Methods For Data Movement

21. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research© 2024 Enterprise Management Associates, Inc. 21
24%
26%
33%
35%
39%
42%
48%
58%
0% 10% 20% 30% 40% 50% 60% 70%
PGP (Pretty Good Privacy) encryption
Removable media encryption
Virtual machine encryption
Full disk encryption (FDE)
File-level encryption
AES (Advanced Encryption Standard) encryption
Database encryption
Cloud storage encryption
Please select the encryption methods used to encrypt data at rest.
Sample Size = 256
Encryption Methods For Data At Rest

22. @ema _resea rch
| @ema_research
File Transfer
Challenges

23. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research© 2024 Enterprise Management Associates, Inc. 23
Sample Size = 154
1%
1%
7%
45%
45%
1%
1%
5%
32%
61%
3%
6%
41%
50%
4%
41%
55%
0% 10% 20% 30% 40% 50% 60% 70%
Very dissatisfied
Dissatisfied
Neutral
Satisfied
Very satisfied
How satisfied is your organization with the performance and reliability of your current file transfer
solutions?
Integration platform as a service (iPaaS)
Workload automation software with native file transfer
Managed file transfer software
FTP servers
Satisfied or Very Satisfied 96%

93%
91%
90%
Satisfaction Of File Transfer Solutions

24. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research© 2024 Enterprise Management Associates, Inc. 24
Sample Size = 154
23%
25%
27%
27%
37%
0% 10% 20% 30% 40%
File size and formats
Costs and licensing
Integration and compatibility
Monitoring and visibility
Security concerns (encryption,
key management)
Top 3 challenges with FTP servers.
25%
26%
26%
32%
33%
0% 5% 10% 15% 20% 25% 30% 35% 40%
File size and formats
Integration and compatibility
Complex workflows and processes
Costs and licensing
Security concerns (encryption, key
management)
Top 3 challenges with managed file transfer (MFT)
software.
25%
26%
27%
28%
28%
0% 5% 10% 15% 20% 25% 30% 35% 40%
Mobile and remote access
Integration and compatibility
Security concerns (encryption, key
management)
Regulatory compliance
Scalability
Top 3 challenges with native file transfer features of
WLA software.
21%
23%
23%
28%
30%
0% 5% 10% 15% 20% 25% 30% 35% 40%
User adoption and training
Regulatory compliance
Costs and licensing
Security concerns (encryption, key
management)
Integration and compatibility
Top 3 challenges with iPaaS software.
Top Challenges For File Transfer Solutions

25. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
iPaaS is quite popular
© 2024 Enterprise Management Associates, Inc. 25
9%
13%
17%
17%
45%
0%5%10%15%20%25%30%35%40%45%50%
Unsure
Yes, add iPaaS
Yes, explore alternative solutions
No, maintain current usage
Yes, expand iPaaS usage
Plans to add or expand iPaaS usage or explore alternative
integration solutions?
Sample Size = 264
45% of those using iPaaS are planning expansion of use, while 13% will add iPaaS and
17% plan to explore alternative solutions.

26. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
Changing File Transfer Solutions
© 2024 Enterprise Management Associates, Inc. 26
14%
17%
13%
18%
33%
5%
0% 5% 10%15%20%25%30%35%
Yes, we are actively looking
Yes, in the next 12 months
Yes, in 12 to 24 months
We are considering a change
No, we are not considering a change
Don’t know
Is your organization considering switching to a new provider for
file transfer solutions?
Sample Size = 264
With data growth, operational challenges, and digital transformation it should be no surprise that
62% of respondents are considering switching to new file transfer software.

27. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research© 2024 Enterprise Management Associates, Inc. 27
6.8%
9.3%
8.1%
9.1%
8.6%
10.1%
10.7%
8.4%
9.4%
10.1%
9.4%
6.1%
6.8%
7.4%
8.0%
8.5%
8.7%
9.5%
9.6%
10.9%
11.0%
13.4%
0.0% 2.0% 4.0% 6.0% 8.0% 10.0% 12.0% 14.0% 16.0%
Middleware and message queue systems
In-house-developed scripts or applications
Enterprise service bus (ESB) and integration platforms
API management platforms
Enterprise resource planning (ERP) systems
Workload automation (WLA) software (using native MFT features)
Managed file transfer (MFT) software
Automated workflow and business process automation tools
Secure file sharing and collaboration tools
Integration platform as a service (iPaaS)
File transfer protocol servers
Distribution of Data Moved in a Typical Month: 2024 vs, 2027 Expectations
20242027
Sample Size = 264
Forecasted Changes In Data Moving Methods
2024 vs 2027

28. @ema _resea rch
| @ema_research
DataOps
Data Pipelines

29. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
DataOps has Grown in Popularity and Adoption Since the Term Was
Coined in 2014
© 2024 Enterprise Management Associates, Inc. 29
33%
46%
20%
1%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Fully implemented and
integrated
Partially implemented,
with ongoing
integration efforts
Initial stages of
implementation, with
plans for further
integration
Not implemented, but
under consideration for
future adoption
How would you rate the level of implementation and integration of
DataOps practices within your organization's data management
processes?
Sample Size = 264
Just 33% of respondents have DataOps fully implemented, with 46% partially implemented.

30. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
DataOps Includes Data Pipelines
© 2024 Enterprise Management Associates, Inc. 30
1%
7%
43%
49%
0% 10%20%30%40%50%60%
Never heard the term "data pipeline"
used in our operations
Not extensively utilized or explored
Used in specific departments or projects
Widely adopted across various
departments and use cases
How extensively are data pipelines utilized within your
organization for data integration and processing?
Sample Size = 264
49% of respondents have widely adopted Data Pipelines.
Another 43% use data pipelines for specific projects or departments.

31. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
Challenges Orchestrating Data Pipelines
© 2024 Enterprise Management Associates, Inc. 31
Sample Size = 235
29%
32%
34%
35%
49%
0% 10% 20% 30% 40% 50% 60%
High operational costs and resource constraints
Compatibility issues with legacy technologies
Integration complexity with existing systems
Scalability and performance issues
Security and compliance concerns
What are the primary challenges your organization faces in orchestrating data
pipelines?
Security and compliance (49%) is the biggest challenge with data pipelines.

32. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
Automation improves data pipelines.
© 2024 Enterprise Management Associates, Inc. 32
36%
44%
47%
49%
53%
61%
0% 10% 20% 30% 40% 50% 60% 70%
Enhanced fault tolerance with built-in fault detection
and recovery mechanisms
Simplified workflow management, scheduling, and
dependency handling
Provided visibility and monitoring capabilities across
the data pipeline
Enhanced scalability and performance
Improved reliability and reduced errors
Increased efficiency and productivity
How has the implementation of automation and orchestration
improved your organization's data pipeline processes?
Sample Size = 217

33. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research© 2024 Enterprise Management Associates, Inc. 33
6%
51%
41%
1%
0%
10%
20%
30%
40%
50%
60%
Manual processes
dominate, requiring
more automation
Partially automated
with room for
improvement
Fully automated with
robust orchestration
capabilities
I don't know
To what extent are data pipelines automated and orchestrated
within your organization?
Sample Size = 235
Automation Is Not Fully Mature In Data Pipelines
Just 41% have fully automated data pipelines, while 51%
have partially automated data pipelines.

34. @ema _resea rch
| @ema_research
@ema _resea rch
| @ema_research
The Data in Motion Report is now available!
Learn more and download:
https://bit.ly/data-in-motion-report
© 2024 Enterprise Management Associates, Inc. 34