Introduction to Apache Solr

«مقدمه‌ای بر آپاچی سولار» ارائه شده توسط حسن نصر در اسکیل‌آپ هشتم

1. Introduction to
Apache Solr
Hassan Nasr Esfahani

2. Topics
–What we need from a text search engine
–What is Solr?
–Why Solr?
–Concepts And Architecture
–Usage
–Special Features
–Competitors

3. Text Retrieval vs Database
Retrieval
–Information and Query
–Unstructured vs Structured
–Ambiguous vs Well defined
–Answers
–Relevant documents (ambiguous) vs matched
documents

4. What we want from text search
engine
Basic Search Features:
–Store some documents with some fields
–Query for documents
Text Search Features
–Find most relevant docs
–Handle Natural language Complications (stop words, stem, tokenizing … )
–Highlight text
–…

5. Problems with Text Search
SampleProblem
مور‌یمقداص‌دمحم‌،‌شباتک‌،Tokenization
ي‌و‌یDifferent Letter representation
دوریم‌،یوریم‌،‌موریمSimilar words
راگزومآ‌و‌ملعمSynonymous words
ریشWord ambiguity
‌،تفر‌،هب‌،‌تسا‌،اب...Stop words
شراذگSpell errors
نونSpoken language

6. What is Solr?
–An Open Search Engine
–Written in Java
–Wrapping Apache Lucene
–With REST API
–Fault tolerant
–Scalable
–Distributable

7. SolrSimple Architecture
Apache Lucene
Query Documents
Analyze
Queue
1
2
3
Analyze
Queue
1
2
3’
Schema.
xml

8. SolrFeatues
–Advanced Search Method
–Language knowledge
–Scoring/Boosting
–Grouping
–Highlighting
–Nested Documents
–Realtime index update

9. How It Works
–SolrServer Contains Some Core ( similar to datebasein
DBMS )
–Each Core specified by schema.xml + …
–Fields
–Data Types
–Analyzers

10. Field List
Field Attributes:
Type
Indexed
Stored
Multivalued
…

11. Data Types
–Int, float , long, double
–Date
–String
–Text ( configurable )
–Location

12. Communicating with Solr
–REST API
–Client Libraris
–JAVA
–Ruby
–PHP
–C#
–Python
–…
–Data Import Handlers
–Direct SQL query

13. Query Format
–DirfferentQuery Parsers:
–Standard(Lucene)
–Dismax
–Edismax
–Block Join Query Parser
–…

14. Standard Query Format
–field:Value
–Phrase search : field:"wordlist"
–Wildcard search : wor?d, word*
–Fuzzy Searches : roam~ matches all terms like foam or foams (max 2 edit distance)
–Proximity Searches (words with maximum distance): "jakartaapache"~10
–Range Searches: [52 TO 10000]or {Aida TO Carmen}
–Bossting: jakarta^4 apache
–Boolean Operators : AND (&&) , OR(||) , NOT(!) , + , -, ( )
–Filter Query

15. Relevancy
–∝Term Frequency
–∝Inverse Document Frequency
–Query Expansion
–…

16. Weakness
–No Transactions
–No join query
–Use as secondary database
–No partial record modification

17. Alternative
–Elasticsearch based on Search
–Mostly towards Analytic Usage
–More popular
–Easier to start
–Less Documented