Collocated Indexing and Distributed Search with GigaSpaces

Compass 2.0 already comes with an integration between GigaSpaces and Lucene/Compass. The integration allows to store the Lucene index (using Lucene Directory) abstraction on top of GigaSpaces. It also allows to use GigaSpaces mirror services allowing to automatically index the content of the data grid (Space) in an asynchronous reliable manner.

The current integration provides great value for both Lucene and Compass users, and it is already used by several users. But, there is still something missing. A big part of GigaSpaces data grid solution is the ability to run collocated services within cluster members, and query them in a distributed manner. This type of integration fits very nicely to the indexing and search requirements by Compass users. Here is a diagram of how this should work:

Lets explain how the integration works by following the flow of operations. The first operation is writing POJOs to the Space (data grid):

1. A POJO is written to the Space using a clustered proxy of the Space.


2. The POJO is directed to one of the partitions (automatically by GigaSpaces).


3. The POJO is written to the collocated cluster member.


4. The indexing service is notified that something changed in the Space (based on POJO templates).


5. The indexing services applies the changes (assuming they have Compass mappings) using Compass API.


6. Compass applies the change (save/delete). The Compass instance stores the Luceen index on the collocated Space (the partition, which might or might not have a backup).

The search operation works as follows:

1. Using a simple POJO search service (CompassSearchService) the search is performed on the clietn side.


2. The search is automatically broadcast to all active partitions and executed on each in an embedded manner.


3. Results that match the query are returned to the client from each partition.


4. The internal client implementation automatically reorders and reduce the results and returns them to the client.

This type of integration takes collocation of indexing and searching to a new level. Indexing and Search operations are performed in a collocated manner in memory making them extremely fast. Scalability is easily handled by adding more partitions, and high availability is provided by adding backups to each partition.

The integration itself can be used both by Compass users and GigaSpaces users. Compass users can use the integration to scale their search integration (use GigaSpace API to store/delete the domain model, and the search API to search). GigaSpaces users can, seamlessly, add google like search support to their current integration with GigaSpaces.

More information can be found in the reference docs (here is a link to the nightly build docs) and this feature is available from tonight nightly builds. The feature in Compass Jira is CMP-666, muhahaha :).