Securing Search Data in the Cloud

Sameer Maggon from Measured Search and Harry Ochiai from Hitachi Solutions discuss securing search data in the cloud.

1. Securing Search Data in the Cloud
Sameer Maggon
Measured Search
Harry Ochiai
Hitachi Solutions
New York Enterprise Cloud Meetup
Jan 25, 2017

2. 2
Agenda
•About the Speakers
•About Measured Search & Hitachi Solutions
•What is Apache Solr?
•Where is Apache Solr used?
•How Search Data is stored
•Data Security Challenge in the Cloud
•Protecting Confidential Search Data
•Challenges of Encrypted Search Index
•Encryption Solution
•Demo
•Q&A
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

3. 3
About the Speakers
Harry Ochiai
•
Senior Business Development Manager of Hitachi Solutions
•
Worked on networking, cyber security, and storage
•
Focus on cloud encryption solutions since 2013
•
New Yorker
Sameer Maggon
•
Founder / Technologist at Measured Search
•
Been working in Open Source Search since 2001 (Lucene/Solr/Elastic)
•
USC Engineering Alumni
•
Works and Lives in Los Angeles, CA
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

4. 4
About Hitachi Solutions
Global IT Solutions Company
•
A Hitachi Company
•
Japan(HQ), North America, Europe, China, India and Southeast Asia
•
12,000 Employees
Leading security solution provider in Japan
•
Innovator and leading provider of encryption technology for over 20 years
•
HIBUN: 40% market share in the endpoint encryption segment in Japan
•
Launched new security solution Credeon globally in 2013
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

5. 5
About Measured Search
Measured Search® enables companies to elevate the experience of Search
based applications faster and with more confidence.
Managed Services &
Support
SearchStax®
Platform as a Service
On-Demand
Expertise & Consulting
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

6. 6
by Measured SearchSearchStax
®

SearchStax
®

Solr Cloud Manager
SearchStax
®

Pulse
SearchStax
®

Analytics
Comprehensive Solr Monitoring &
Alerting with service level reporting
to proactively manage your clusters.
Realtime feedback & user
insights to help optimize
your Search Experience
Easiest way to run & manage
Solr in the cloud - saves time,
money and reduces risk.
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

7. 7
What is Apache Solr?
Solr is the popular, blazing-fast, open source
enterprise search platform built on Apache
Lucene™
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

8. 8
Where is Apache Solr used?
Government
eCommerce
Education
Life Sciences
Entertainment
Healthcare
Financial Services
High Tech
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

9. 9
Where is Apache Solr used?
finding tickets
finding job
finding restaurant/services
Enterprise
Search
Media
Search
Retail
Customer
Search
Fraud
Analytics
Publishing
RecruitingTravelResearch
Business
Intelligence
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

10. 10
Search Index
Search platforms maintains internal indices of terms and properties of each
indexed document in plaintext.
Plain Search Index Encrypted Search Index
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

11. 11
Data Security Challenges in the Cloud
Solr / Search Cluster
Backups
Threats
Managed Service Provider (MSP)
Rogue Employee
Managed Service Provider Rogue Employee Hacker Accidental Data Access
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

12. 12
Challenges of Encrypted Search Index
Challenges
•
To search through encrypted data, data must be decrypted
•
Decryption slows down the process
•
Encryption limits usability
Goals
•
Maintain encrypted state without sacrificing security level
•Encrypt using your own key
•
Maintain search performance and usability
•Protect against unauthorized users and rogue system administrators
•Regulatory compliance
Solution
•
Searchable Encryption technology
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

13. 13
Protecting Confidential Search Data
Solr/Lucene
textKey
word
result
value
Solr/Lucene
textKey
word
result
File System
Encryption
value
Solr/Lucene
textKey
word
result
value
Simple
Encryption
Solr/Lucene
textKey
word
result
value
Searchable
Encryption
textKey
word
result
value
Searchable
Encryption
Storage Storage Storage Storage Storage
Client
Server
App
Server
OS
No Encryption OS Encryption
Simple Encryption
with Solr Plugin
Client-Side Searchable
Encryption
Searchable Encryption
with Solr Plugin
Low Security High Security
No Security Decryption at storage layer
XDifficult to separate key
Decrypt first and match
XVery slow
Xplaintext in memory
Match first and decrypt
OKey separation
OHigh Performance
(1)
OSemantically Secure
(2)
Xplaintext in memory
Client Client Client Client Client
Decryption at client-side
OKey separation
OHigh Performance
(1)
OSemantically Secure
(2)
Ono plaintext on server
Solr/Lucene
(1)Use of Symmetric Key
(2)Probabilistic Encryption Scheme
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

14. 14
Search Encryption 101
Searchable Encryption is an encryption technology for searching data in an
encrypted state.
•
Fully Homomorphic Encryption
•
Homomorphic Encryption
•
Functional Encryption
•
Hitachi Searchable Encryption
Practical
CKA
CPA
Tokenization
10
3
10
6
10
0
10
-3
Hitachi
Searchable
Encryption
Functional
Encryption
Homomorphic
Encryption
Fully Homomorphic
Encryption
Security
Performance (Search / sec)
Encrypted
Plain
CKA: Chosen Keyword Attack
CPA: Chosen Phrase Attack
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

15. 15
Search Encryption 101
Encryption Schemes
•
Deterministic
•
Constant value
•
Vulnerable to statistical attacks
•
Probabilistic
•
Random value
•
Semantically secure
Encryption Key Exchange
•
Symmetric
•
Asymmetric / PKI
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

16. 16
Solution: Searchable Encryption with Solr Plugin
Hitachi Credeon Secure Full-Text Search
•Searchable Encryption plugin for Apache Solr and Elasticsearch
•
Deterministic Encryption Scheme
•
128 bit randomization
•
AES 256, FIPS 140-2
•Symmetric Key
•Real-time search (15%+ overhead)
•
Key Management System, Java KeyStore
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

17. 17
Solution: Client Side Searchable Encryption
Credeon Secure Document Solution for SharePoint Online
•Client-side encryption for search index and data
•Searchable encryption on Solr
•
Search Engine and Key Management are independent of Microsoft
Search Server
SharePointServer
Key Management
Server
Search Engine
SharePoint Online
Client PC
1. Get a key
2. Index the file contents
and encrypt index
4. Upload encrypted
file
4. Upload encrypted
index
3. Encrypt file
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

18. 18
Demo: Securing Solr Search in the Cloud
SearchStax with Credeon
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

19. 19
Q&A
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.

20. 20
Contact Info
Sameer Maggon
@maggon
[email protected]
https://www.measuredsearch.com
Harry Ochiai
@credeon
[email protected]
https://psg.hitachi-solutions.com/credeon/overview
© Hitachi Solutions America, Ltd. and Measured Search, Inc. 2017, All rights reserved.