Elastic enhances ES|QL with LOOKUP JOIN & cross-cluster search
Elastic has introduced a range of new features to its Elasticsearch Query Language (ES|QL), aimed at increasing the efficiency of querying large, distributed datasets.
The latest updates, available in versions 8.19 and 9.1 of Elasticsearch, incorporate the general availability of LOOKUP JOIN and Cross-Cluster Search (CCS), more than 30 performance enhancements, and additional settings for improved resilience and observability.
These changes are designed for organisations dealing with high-volume workloads in observability, security, and operational data environments.
ES|QL is actively used on over 10,000 clusters each week, supporting enterprise-scale search and analytics. With these enhancements, the language is positioned to meet the demands of data-rich industries.
Major enhancements
One of the headline additions is LOOKUP JOIN, which is now generally available. This feature allows users to enrich and correlate data across indexes without the need to denormalise data or manage joins on the client side.
Scenarios supported include merging disparate sets of data such as security logs with employee directories or integrating threat intelligence information. This is achieved within the ES|QL framework by using a single piped query structure.
The expanded functionality of LOOKUP JOIN includes mixed-type joins, which permits joins between compatible numeric fields (such as long and integer types), and support for index aliases.
This provides users with more flexibility and reduces query complexity. Additionally, high-precision joins are now possible through support for date_nanos, which is relevant for applications requiring high-frequency or financial-grade data analysis.
Distributed query execution
ES|QL now includes general availability for Cross-Cluster Search, extending its capability to run queries over distributed Elasticsearch clusters located in different regions or environments. This upgrade is intended to break down data silos and enable analysis across various types of telemetry and workloads, such as observability and security, on a global scale.
Ajay Nair, General Manager, Platform at Elastic, commented on the release:
"With today's release, ES|QL becomes even more powerful, observable, and fault-tolerant out of the box. Whether you're correlating live security data or running distributed queries across global clusters, these enhancements help developers move faster with more confidence."
Improvements in resilience and fault tolerance
The new allow_partial_results setting, which is now enabled by default, lets queries complete even if some shards are temporarily unavailable. This increases system resilience, enabling operations to continue through transient errors or outages. Additionally, ES|QL can now automatically retry failed shard operations, reducing query failures during processes such as rolling upgrades or isolated node issues.
Query observability and performance
Elastic has also focused on observability with fresh query monitoring features.
The system now logs all ES|QL queries, a development that aids in usage analysis and troubleshooting. There's a new live query monitoring API (in technical preview) that enables users to view running queries and examine profiling data for debugging and optimisation purposes.
The software releases feature over 30 performance and resource utilisation enhancements.
These include more effective query planning, prioritisation of frequently accessed ('hot') data, and a suite of memory and CPU usage improvements affecting ES|QL commands such as REPLACE and TO_IP.
One specific optimisation involves more aggressive "pushdowns" to the Lucene search library, with speed increases of up to 86 times for certain filtering operations.
Addressing enterprise requirements
These capabilities are intended to benefit organisations managing substantial data volumes and requiring dependable, fast querying for their observability and security workloads.
By enabling live data correlation, distributed search across clusters, and enhanced performance, the ES|QL improvements align with requirements around stability and operational efficiency over petabyte-scale datasets.
