Searching Across Indexes and Types

In this article, I will show you how you can search across indexes and mapping types, rather than having to explicitly define which index and mapping type to search.

Before showing you examples of how to do this, I will just add a new index and a new mapping type for demonstration purposes, so that I have some meaningful to search for. The last time I added an index, I did this explicitly, but I also mentioned that an index can be created implicitly by simply adding a new document to a non-existing index. So to demonstrate this approach, I will just be adding a new document to an index that does not yet exist.

Let’s say that we are building a blog for recipes with a sample name of myfoodblog. I will add an index of that name with a recipe mapping type. I will just paste in a PUT request that adds a recipe document into an index with the name of myfoodblog. The document just has the fields name, description and ingredients. So let’s just issue the request.

PUT /myfoodblog/recipe/1
{
  "name": "Pasta Quattro Formaggi",
  "description": "First you boil the pasta, then you add the cheese.",
  "ingredients": [{
    "name": "Pasta",
    "amount": "500g"
  }, {
    "name": "Fontina cheese",
    "amount": "100g"
  }, {
    "name": "Parmesan cheese",
    "amount": "100g"
  }, {
    "name": "Romano cheese",
    "amount": "100g"
  }, {
    "name": "Gorgonzola cheese",
    "amount": "100g"
  }]
}

This request is conceptually identical to the request I issued in a previous lecture where I added a product to the ecommerce index. The difference compared to this request is what happens implicitly behind the scenes. In the previous lecture, we had already created the ecommerce index and defined the product mapping type explicitly. In this case, I had not created the index nor the mapping type in advance, but this is done automatically by Elasticsearch as a result of this request. A recipe mapping type is created and the field data types are inferred based on the document’s data through dynamic mapping, which you might recall me explaining earlier in the course.

Let’s verify that the index has been created by listing all of the indexes, using the same command as previously.

GET _cat/indices?v

As you can see in the results, the myfoodblog index has been created and it contains one document. Next, let’s check the mapping information that Elasticsearch automatically inferred for the recipe mapping type, based on the data in the document that I added.

GET /myfoodblog

As you can see from the results, Elasticsearch has determined that all of the data that I added were strings, and that the ingredients field was a nested field.

Now that I have documents in two indexes and two mapping types, I am ready to perform a search that searches across indexes and mapping types.

GET /ecommerce,myfoodblog/product/_search?q=pasta

This query operates on both the ecommerce and myfoodblog indexes and looks for a document with a value of pasta and a type of product. I used the query string API for convenience, because the important part for this lecture is all in the URL either way. As you can see, the way to search multiple indexes is to conveniently separate each index with a comma, so in this case, I am searching the ecommerce and myfoodblog indexes. Although I am searching both of my indexes, I only get matches in the ecommerce index, which is because I have no matching documents within a product type for the myfoodblog index.

Now that I have shown you how to search multiple indexes, the next thing I will show you is how to search multiple mapping types. As you may have guessed, this is also done by separating the types by a comma, so I will just add the recipe type to the query.

GET /ecommerce,myfoodblog/product,recipe/_search?q=pasta&size=15

Note that I have added a size parameter. The reason for that is that I have 10 matching documents within the ecommerce index and one within the myfoodblog index. Since the default result set size is 10, the document from the myfoodblog index would be excluded due to how the documents are scored. Therefore I have increased the size of the result set to 15 just so that I can show you matching documents from both indexes. I will discuss the size parameter in a later lecture, so don’t pay attention to that for now. The results contain documents from both indexes.

Something to note is that when searching multiple indexes and mapping types, the _index and _type meta fields are quite useful. This is because they enable you to know which index or type a given document belongs to. Say, for instance, that there are different display rules for products and recipes. By inspecting these meta fields, you could add different logic based on a document’s mapping type, for example.

There is another small feature that I want to mention. It is possible to include and exclude indexes by prefixing index names with plus or minus signs, respectively. I will show you how to do this by typing in a query that is very similar to the previous one.

GET /-ecommerce,%2Bmyfoodblog/product,recipe/_search?q=pasta

In this query, I am excluding documents that are stored within the ecommerce index by prefixing the index name with a hyphen. You are probably wondering why I prefixed the myfoodblog index with %2B. I did this because in URLs, a plus sign is interpreted as a space, so to avoid this, I URL encoded the plus sign. If I did not do this, Elasticsearch would interpret the plus sign as a space and return an error. The results of this query will be documents of the types product and recipe within the myfoodblog index that match the term pasta.

Lastly, I quickly want to show you a few queries, the first one being how to search for all types.

GET /ecommerce/_search?q=pasta

By simply leaving any type out of the URL, Elasticsearch will search for any type. In this case, any type may match within the ecommerce index. Earlier in this lecture, I showed you how to search specific indexed. It is also possible to search all of the indexes within a cluster by using the _all placeholder.

GET /_all/product/_search?q=pasta

In this example, I am searching for documents of the type product within any index in the cluster. If I wanted to search for any type, I could simply remove the type from the URL, or I could simplify the query.

GET /_search?q=pasta

By simply leaving out any information on which indexes or types to search, Elasticsearch will search across all indexes and all types.