Elasticsearch (Search)

Theory

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. It stores JSON documents in indices, inverted indexes every analyzed field for fast full-text search, and scales horizontally via sharding.

Inverted index: maps each term → list of document IDs containing it. At search time, query terms lookup posting lists and intersect them — O(1) term lookup vs scanning every document. This is why full-text search on billions of docs is feasible.

Analysis pipeline: Character filters → Tokenizer (splits text) → Token filters (lowercase, stop, stem). Analyzer at index time and search time must match for relevance. Custom analyzers combine tokenizer + filters in index settings.

Query DSL: match analyzes input then searches (full-text). term queries exact values on keyword fields (no analysis). bool combines must/should/must_not/filter. filter context scores 0 but caches bitsets — use filter for yes/no conditions.

BM25 is the default relevance scoring — considers term frequency, inverse document frequency, and field length normalization. Use the explain API to debug ranking. Boost fields or queries to tune relevance for your domain.

Index settings: number_of_shards (fixed at creation), number_of_replicas (HA + read scaling). Target shard size 10–50GB. Too many small shards waste overhead; too few limits parallelism. Replicas serve search traffic and failover.

Aggregations: bucket (terms, date_histogram) group documents; metric (avg, sum, cardinality) compute statistics. Nested aggregations drill down — e.g. terms on category then avg price per bucket. Pipeline aggs operate on other agg outputs.

Index Lifecycle Management (ILM) automates hot/warm/cold/delete phases. Data streams with @timestamp simplify log ingestion — rollover creates new backing index when size or age threshold is hit. Delete phase enforces retention without manual cleanup.

Architecture Diagram

Users / clients
         |
  Elasticsearch (Search)
         |
  Core services
         |
  Data + observability

Examples

PUT /products
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "analysis": {
      "analyzer": {
        "product_name": {
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": { "type": "text", "analyzer": "product_name",
                 "fields": { "keyword": { "type": "keyword" } } },
      "price": { "type": "float" },
      "created_at": { "type": "date" }
    }
  }
}

GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "title": "wireless keyboard" } }
      ],
      "filter": [
        { "range": { "price": { "lte": 100 } } },
        { "term": { "in_stock": true } }
      ]
    }
  },
  "sort": [{ "price": "asc" }],
  "size": 20
}

GET /logs-*/_search
{
  "size": 0,
  "aggs": {
    "errors_over_time": {
      "date_histogram": { "field": "@timestamp", "fixed_interval": "1h" },
      "aggs": {
        "by_service": {
          "terms": { "field": "service.keyword", "size": 10 },
          "aggs": {
            "avg_latency": { "avg": { "field": "duration_ms" } }
          }
        }
      }
    }
  }
}

Key Concepts

Interview Questions

Best Practices

• Define explicit mappings before indexing — dynamic mapping guesses wrong types (e.g. IP as text, dates as strings).
• Use keyword subfield for exact filters and aggregations — text fields are analyzed and unsuitable for sorting/terms.
• Size shards for retention, not vanity — target 10–50GB per shard; too many small shards wastes heap.
• Use filter context in bool queries for non-scoring clauses — filters are cached and faster.
• Avoid deep pagination (from: 10000) — use search_after for cursor-based paging.
• Set index lifecycle policies (ILM) — roll hot → warm → delete to control cluster disk growth.

Common Mistakes

• Mapping a field as text then aggregating on it — aggregations require keyword or numeric types.
• Default 5 primary shards on tiny indices — 1GB index with 5 shards wastes overhead on a 3-node cluster.
• Indexing without specifying idempotency — re-indexing duplicates documents unless using external _id.
• Running wildcard queries on leading *term — scans all terms in the index, extremely slow at scale.
• Ignoring cluster yellow/red state — unassigned shards mean data loss risk or capacity exhaustion.

Cheat Sheet

Practical Exercises