Elasticsearch Optimization: Shards, refresh_interval, Queries

We integrate Elasticsearch optimization disciplines to resolve performance issues without requiring hardware upgrades. Too many shards kill performance more reliably than weak CPUs. An overly frequent refresh makes indexing 3–5 times slower than necessary. We have been optimizing Elasticsearch for our clients for over 5 years — in our experience, 90% of issues are solved by tuning, not by server upgrades. Elasticsearch optimization is primarily about proper design, then hardware tuning.

How to properly configure the number of shards?

Each shard is a separate Lucene index instance (Elasticsearch Documentation) with its own file descriptors, JVM objects, and heap overhead. On a 5-node cluster, keeping 500 small indices with 50 shards each = 25,000 shards = a crawling cluster. Rule of thumb: 1 shard = 10–50 GB of data. Smaller — too many shards (overhead dominates data). Larger — difficult to rebalance when adding a node. Maximum shards per 1 GB heap: ~20 shards. For a 16 GB heap, no more than 320 shards per node.

Check shard statistics with _cat/shards and _cat/nodes. Reducing shard count via the Shrink API requires disabling writes and moving all shards to one node, then executing _shrink.

Why is refresh_interval so important?

Elasticsearch by default refreshes every second — creates a new Lucene segment from the in-memory buffer and makes documents searchable. Each refresh involves file operations, segment creation, and IO load. For real-time search (chat, notifications), keep 1s. For analytics, logs, ETL, increase to 30s–300s. During bulk data loading, disable refresh temporarily: "index.refresh_interval": "-1". Speed gain: 3–5x.

Merge Policy and forcemerge

Lucene periodically merges small segments into larger ones (merge). This frees space from deleted documents and speeds up search. Using a tiered merge policy with appropriate parameters reduces IO overhead. For read-only indices (archived data, completed rolling indices), force merge to 1 segment:

POST /logs-archive/_forcemerge?max_num_segments=1 

After forcemerge, search is significantly faster, and size decreases by 20–40% due to removal of tombstone records. Do not run forcemerge on actively indexed indices — it creates huge IO load.

Replicas and Bulk API

A replica is a synchronous copy of a shard on another node. When bulk-loading data into a new index, temporarily disable replicas: "index.number_of_replicas": 0. Speed gain: 2–3x with 1 replica, 3–4x with 2. Bulk API — avoid indexing documents one by one. Use parallel loading with a batch size of 5–15 MB. Example in Python with parallel_bulk:

from elasticsearch import Elasticsearch from elasticsearch.helpers import parallel_bulk es = Elasticsearch([...]) def generate_actions(data): for item in data: yield {"_index": "products", "_source": item} for ok, info in parallel_bulk(es, generate_actions(data), chunk_size=500, max_chunk_bytes=10*1024*1024): if not ok: print(info) 

Query Optimization

Filter vs. Query: use filter wherever scoring is not needed. Filters are cached at the shard level — filter queries are up to 3 times faster than equivalent query queries. Wildcard and regexp are expensive, especially with leading wildcards. Replace with edge N-grams. Deep pagination: from: 10000 is costly. Use search_after with sorting. For aggregations, leverage doc_values to reduce memory pressure.

Monitoring and GC

Profile API — detailed query breakdown. Hot Threads API — what the JVM is doing. When heap > 85%, aggressive G1GC kicks in and queries slow down. Configure jvm.options for G1GC: -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30.

What’s included in the audit and optimization

• Analysis of current cluster configuration (shards, replicas, refresh_interval, merge policy)
• Load testing with query profiling
• Recommendations on sharding and refresh_interval tuning
• Query optimization using filter, search_after
• G1GC and heap tuning
• Documentation of all changes
• Access to your cluster for analysis
• A training session for your team
• 30 days of ongoing support

Contact us for an evaluation of your cluster — our certified engineers with 5+ years of experience guarantee 2–5x search speedup. Order an Elasticsearch performance audit and receive a detailed report with recommendations. Infrastructure savings can reach 30% — for a typical 5-node cluster costing $1,700/month, that's $500 saved monthly.

Elasticsearch Documentation — official performance tuning guide.