Red Hat JBOSS Data Virtualization

Red Hat JBOSS Data Virtualization

1. www.dlt.com
Red Hat JBoss Data Virtualization
July, 2016
Rick Stewart, Middleware SA
Herndon, VA

2. 7/19/16 DLT Solutions LLC –Proprietary & Confidential 2
“Kiss” “Whitesnake” “Poison”
“Bad Company”
Data
Warehouse

3. “Bad Company”
7/19/16 DLT Solutions LLC –Proprietary & Confidential 3
“Kiss” “Whitesnake” “Poison”
Data
Warehouse
Data Virtualization Server

4. What does Data Virtualization software do?
7/19/16 DLT Solutions LLC –Proprietary & Confidential 4
Virtual Consolidated Data Source
BI Reports
Data Virtualization Software
•Consume
•Compose
•Connect
SAP Salesforce.comOracle DW XML, CSV
& Excel files
Siloed &
Complex
Virtualize
Abstract
Federate
Easy,
Real-time
Information
Access
Applications
DATA CONSUMERS
DATA SOURCES

5. “Bad Company”
7/19/16 DLT Solutions LLC –Proprietary & Confidential 5
“Kiss” “Whitesnake” “Poison”
Data
Warehouse
Data Virtualization Server

6. “Bad Company”
7/19/16 DLT Solutions LLC –Proprietary & Confidential 6
“Kiss” “Whitesnake” “Poison”
Data
Warehouse
Data Virtualization Server

7. Data Challenges Getting Bigger
7/19/16 DLT Solutions LLC –Proprietary & Confidential 7
BI Reports
Operational
Reports
Enterprise
Applications
Cloud Native
Applications
Mobile
Applications
Hadoop NoSQL Cloud Apps Data Warehouse
& Databases
Mainframe XML, CSV
& Excel Files
Enterprise Apps
Integration Complexity
Consumption & Creation
Siloed
How to Integrate?

8. Improve Access to Your Data
7/19/16 DLT Solutions LLC –Proprietary & Confidential 8
BI Reports
Operational
Reports
Enterprise
Applications
Cloud Native
Applications
Mobile
Applications
Hadoop NoSQL Cloud Apps Data Warehouse
& Databases
Mainframe XML, CSV
& Excel Files
Enterprise Apps
Broad & Streamlined
Adaptable & Secure
Federated & Meaningful
Data Virtualization Server

9. Simplify Access to Your Data
7/19/16 DLT Solutions LLC –Proprietary & Confidential 9
streaming
databases
social
media data
production
application
big data
stores
website
ESB
analytics
& reporting
unstructured
data
mobile
App
data
warehouse
& data marts
internal
portal dashboard
external
data
private
data
ODBC/SQL JDBC/SQL XML/SOAP REST/JSON OData SQL
JMS SQL JDBC OData Hive RSS Excel JSONREST SOAP
JMS message SQL statement SOAP messageData Virtualization Server
production
databases
applications

10. Turn Siloed Data into Actionable Information
7/19/16 DLT Solutions LLC –Proprietary & Confidential 10
Connect
Compose
Consume
BI Reports & Analytics
Mobile Applications
Applications & Portals ESB, ETL
Native Data Connectivity
Standard based Data Provisioning
JDBC, ODBC, SOAP, REST, OData
JBoss
Data
Virtual-
ization
Data
Consumers
Data
Sources
Design Tools
Dashboard
Optimization
Caching
Security
Metadata
Hadoop NoSQL Cloud Apps Data Warehouse
& Databases
Mainframe
XML, CSV
& Excel Files
Enterprise Apps
Siloed &
Complex
Virtualize
Transform
Federate
Easy,
Real-time
Information
Access
Unified Virtual Database / Common Data Model
Data Transformations

11. Supported Data Sources
7/19/16
DLT Solutions LLC –Proprietary & Confidential 11
Enterprise RDBMS:
•Oracle
•IBM DB2
•Microsoft SQL Server
•Sybase ASE
•MySQL
•MariaDB
•PostgreSQL
•Ingres
Enterprise EDW:
•Teradata
•Netezza
•Greenplum
Search:
•Apache SOLR
Hadoop:
•Apache
•HortonWorks
•Cloudera
•More coming…
Office Productivity:
•Microsoft Excel
•Microsoft Access
•Google Spreadsheets
Specialty Data
Sources:
•ModeShape Repository
•Mondrian
•MetaMatrix
•LDAP
•Apache POI for Excel
NoSQL:
•JBoss Data Grid
•MongoDB
•Cassandra
•More coming…
Enterprise & Cloud
Applications:
•Salesforce.com
•SAP
Technology
Connectors:
•Flat Files, XML Files,
XML over HTTP
•SOAP Web Services
•REST Web Services
•OData Services
7/19/16

12. Data As A Service
DLT Solutions LLC –Proprietary & Confidential 127/19/16
Contextual view of disparate
source data
Single point of access
Standard based interfaces
Shareable integration and
transformation logic
Reusable data services
But you cannot achieve this by
writing more application code…
Hadoop
NoSQL Cloud Apps Data Warehouse
& Databases
Mainframe XML, CSV
& Excel Files
Enterprise Apps
JBoss Data Virtualization
BI Dashboard
& Reports
Analytical
Applications
ESB/SOA
Integration
BPM
Applications
Mobile
Applications
SQL Statement SOAP MessageREST Message
REST Request
JSON Result
SQL Request
SQL Result

13. Logical Architecture
7/19/16 DLT Solutions LLC –Proprietary & Confidential 13
Data Consumers
Data Sources

14. Teiid Data Virtualization Designer
7/19/16 DLT Solutions LLC –Proprietary & Confidential 14

15. 7/19/16 DLT Solutions LLC –Proprietary & Confidential 15
Tooling VirtualDBEngine Server

16. 7/19/16 DLT Solutions LLC –Proprietary & Confidential 16
Tooling VirtualDBEngine Server
Users create data models
based on metadata:
•Imported from data
sources
•Supplied via DDL
•Provided by Engine
•Specified by user
Models are packaged in a
Virtual Database (VDB)
Physical Models representing actual data sources
Logical Models

17. 7/19/16 DLT Solutions LLC –Proprietary & Confidential 17
Tooling VirtualDBEngine Server
Build XML Document
models from XML Schemas
Map XML Document
models to other data models
Enable data access via
XML

18. 7/19/16 DLT Solutions LLC –Proprietary & Confidential 18
Tooling VirtualDBEngine Server
Virtual Databases (VDBs) are deployment
archives similar to .WAR.
VDBs contain
•Source metadata and models
•View metadata and models
•System metadata
•Connection information, which is bound to
sources at deployment time
VDBs are deployed to the query engine
VDB Internals
Source Models
Connector
Binding
Properties
View Models
Manifesto Info

19. 7/19/16
19
Tooling VirtualDBEngine Server
JBoss Data Virtualization can offer finer-grained
security control:
Authentication:Kerberos, LDAP, WS-UsernameToken,
HTTP Basic, SAML
Authorization:Virtual data views, Role based access
control
Administration:Centralized management of Virtual DB
privileges
Audit:Centralized audit logging and dashboard
Protection:
Row and column masking
SSL encryption (ODBC and JDBC)
DLT Solutions LLC –Proprietary & Confidential

20. 7/19/16 DLT Solutions LLC –Proprietary & Confidential 20
Tooling VirtualDBEngine Server
Query Engine
JDBC API
VDB
Connector
Binding (1)
Connector
Binding (2)
C1 C2
DB
Oracle
DB
SQL Server
Data Consumer Apps
Query Engine is core data
virtualization functionality: Federating
relational query engine. Rule and cost
based optimizer, advanced query
planner, caching, hint processing.
Query Engine hosts VDBs, binds to
data sources, performs query
execution and results processing.

21. 7/19/16
21
Tooling VirtualDBEngine Server
The Teiid Query engine is hosted in
JBoss EAP and uses key container-
provided services:
•Transaction manager
•JAAS security framework
•Container managed data sources
•EAP management infrastructure
•EAP deployment
The Serverexposes views /services
to consumers and managed
connections and connection pools for
data sources.
DLT Solutions LLC –Proprietary & Confidential
JBoss EAP
Applications
Security
JAAS
Transaction
Manager
JDV Runtime Engine
BufferMgr
Threading
Local Caches
etc.
VDB
VDBs
ODBC Socket
Transport
Admin Socket
Transport
JDBC Socket
Transport
Profile
Service
ODBC
JDBC
Admin /
AdminShell
JON
DS
DS
DS
DS
JCA
Translators
Embedded DS
xxx-ds.xml
yyy-ds.xml
zzz-ds.xml

22. 7/19/16
22
Tooling VirtualDBEngine Server
DLT Solutions LLC –Proprietary & Confidential
CACHING & MATERIALIZATION
Multiple levels of caching to meet
performance requirements and manage load
on source systems:
Materialized Views
–External or Internal materialized views
–Ability to override use of materialized
views
Result set Caching
–Applied to results return from user queries
and virtual procedure calls
–Configurable time to live and max. number
of entries
Code Table Caching
–Suited for integrating reference data with
transaction/operational data e.g. Country
code, State Code etc.
QUERY
Access Patterns –criteria requirements on
pushdown queries
Pushdown –decompose user query into
source queries
–Projection minimization to remove unused
select items
–Decompose aggregates over joins/unions
–Generating SQL matching Teiid system
functions
Dependent Joins (can use hints) –feed equi-
join values from one side of the join to the
other
Partition aware aggregation and joins
Copy Criteria –uses criteria transitivity to
minimize join tuples.
PERFORMANCE OPTIMIZATION

23. Business Dashboard
7/19/16 DLT Solutions LLC –Proprietary & Confidential 23

24. Bring It All Together
7/19/16 DLT Solutions LLC –Proprietary & Confidential 24
Hadoop
Data Integration
JBoss Data Virtualization
In-memory Cache
JBoss Data Grid
BI Analytics
(historical, operational, predictive)
Composite Applications
Messaging and Event Processing
JBoss A-MQ and JBoss BRMS
J
Structured Data
Streaming
Data
Semi-Structured
Data
Capture & Process
Integrate & Analyze
Red Hat Storage

25. 25
Questions
?

26. Bring It All Together
7/19/16 DLT Solutions LLC –Proprietary & Confidential 26

27. 27
Thank
You!

28. JBoss Data Virtualization –Use Cases
7/19/16 DLT Solutions LLC –Proprietary & Confidential 28
Self-Service
Business
Intelligence
Thevirtual, reusable data model provides business-friendly representation of data,
allowing the user to interact with their data without having to know the complexities of
their database or where the data is stored and allowing multiple BI tools to acquire data
from centralized data layer. Gain better insights from Big Data using JBoss Data
Virtualization to integrate with existing information sources.
360
◦
Unified
View
Deliver a complete view of master & transactional data in real-time. The virtual data layer
serves as a unified, enterprise-wide view of business information that improves users’ ability
to understand and leverage enterprise data.
Agile SOA
Data
Services
A data virtualization layer deliver the missing data services layer to SOA applications. JBoss
Data Virtualization increases agility and loose coupling with virtual data stores without the
need to touch underlying sources and creation of data services that encapsulate the data
access logic and allowing multiple business service to acquire data from centralized data
layer.
Regulatory
Compliance
Data Virtualization layer deliver the data firewall functionality. JBoss Data Virtualization
improves data quality via centralized access control, robust security infrastructure and
reduction in physical copies of data thus reducing risk. Furthermore, the metadata
repository catalogs enterprise data locations and the relationships between the data in
various data stores, enabling transparency and visibility.

29. 7/19/16 DLT Solutions LLC –Proprietary & Confidential 29
BA C D
JBoss Data Virtualization
Leveraged TPC-H like schema, data and queries
Use 4 different commercial enterprise RDBMS
Each database with 1 TB data representing
•150 million customers, with over
•600 million order records, and
•6 billion order line items.
•Total 4 TB of data
Findings:
•No measurable JDV queries overhead vs. direct queries
•Queries to federated data from four data sources ran
61.7 percent faster vs. baseline
•Scaling queries workload by 2x resulted in <10% impact
on response time
Download Benchmark Study @ http://www.redhat.com/en/resources/jboss-data-virtualization-query-performance-benchmark-study