Meta Fields

Now that we have taken a look at field data types, it is time to talk about meta fields.

Introduction to Meta Fields

So, just what are meta fields? Each document that is indexed in Elasticsearch has metadata associated with it. The names of these fields are prefixed with an underscore, and some of them can be customized when creating a mapping type. We will go through the various meta fields below.

Identity Meta Fields

Now we will discuss the various categories of meta fields, the first one being identity meta fields.

_index

The _index field contains the name of the index that a document belongs to and allows for matching documents based on the index that it is stored within. It is actually a virtual field and is therefore not stored as a real field within Lucene.

_type

The _type field stores the mapping type of the document – this could be employee, for instance. This field is indexed to make searching for specific types of documents fast.

_id & _uid

The _id field contains the ID of the document. The field is not indexed, because it can actually be derived from the _uid field, which contains both the document ID and type, separated by a hashtag – {type}#{id}. You are probably not going to make use of this field, because the _id and _type fields are made available to you for convenience, but it is nevertheless always good to know what happens behind the scenes.

Document Source Meta Fields

The next category of meta fields is document source meta fields, which contains two fields.

_source

The _source field contains the JSON that was passed to Elasticsearch at index time, or the updated values if the document has since been updated. The field is not indexed, and therefore it is not searchable, but it is returned in search or fetch results. It can be disabled if you want to save storage space.

_size

The _size field contains the size of the _source field in bytes. However, it’s only available after installing the mapper-size plugin, which can be done with the below command.

sudo bin/plugin install mapper-size

Indexing Meta Fields

The next category of meta fields is indexing meta fields.

_all

The _all meta field contains the concatenated values of all other fields, using a space as the delimiter. It is analyzed and indexed, but not stored, meaning that it can be searched but not retrieved. This field is useful if you want to search documents for a value but do not know or care in which field the value is present.

_field_names

The _field_names field indexes the names of the fields for a document that contains a value other than NULL. This is used by the exists and missing queries to find documents that have or do not have a value for a given field.

Routing Meta Fields

The next category is routing meta fields.

_parent

The first routing meta field that we will talk about, is the _parent field. This makes it possible to define parent-child relationships between documents within an index, by making one mapping type the parent of another mapping type. This is very similar to having a foreign key to the same table within a relational database.

"mappings": {
	"employee": {
		"_parent": {
			"type": "person" 
		}
	}
}

Consider the above example, where the parent type of the employee type is set to the person mapping type. When you add a document to an index, you can specify the parent ID in the URL to establish the relationship for a particular document. There are then a number of queries that can be used to require a given relationship to exist, for example.

_routing

The _routing meta field is used to route a document to a particular shard within an index, and enables you to define rules that control in which shards documents are stored. Chances are that you are fine with the defaults that Elasticsearch provides, but for advanced use cases, this is indeed a possibility. Nevertheless, the chances that you will ever need this are slim, but it is good to know that the possibility exists.

Other Meta Fields

Lastly, there is one meta field that doesn’t fit into any of the other categories, namely the _meta field.

_meta

The _meta field is used to store application-specific meta data, which can be anything you want. This is possible because each mapping type can have custom meta data associated with it, and Elasticsearch does not do anything with this meta data except storing and retrieving it.

That’s it! That concludes the theoretical part of mapping in Elasticsearch. It is now time to make use of this theory and add a mapping to our index in the next article.