[[ilm]]
[role="xpack"]
== Set up index lifecycle management

beta[]

You can use the {ref}/getting-started-index-lifecycle-management.html[index
lifecycle management] feature in {es} to manage your {beatname_uc} indices as
they age. For example, instead of having {beatname_uc} create daily indices
where index size can vary based on the number of Beats and number of events
sent, you can use an index lifecycle policy that automates a rollover to a new
index when the existing index reaches a specified size or age. 

{beatname_uc} provides a default policy that you can load when you set up
{beatname_uc}. The default policy is applied to any new indices created by
{beatname_uc}. You can edit the policy to modify the lifecycle of both new and
existing indices.

To use index lifecycle management on {beatname_uc} indices:

. Enable index lifecycle management by setting `ilm.enabled: true` in the {es}
output configuration. For example:
+
--
[source,yaml]
------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["localhost:9200"]
  ilm.enabled: true
------------------------------------------------------------------------------

This configuration overwrites your index settings and adjusts the {beatname_uc}
template to use the correct settings for index lifecycle management.

NOTE: If you've previously loaded the index template for this version into {es}, 
you must overwrite the template by setting `setup.template.overwrite: true`.

The rollover alias is set to +{beatname_lc}-\{beat.version\}+ by default. You
can change the prefix used in the alias by setting `ilm.rollover_alias`, but you
can't remove `{beat.version}` from the rollover alias name. The default pattern
used for the rollover index is `%{now/d}-000001`. You can change the
pattern by setting `ilm.pattern`. For example:

["source","yaml",subs="attributes"]
----
output.elasticsearch:
  hosts: ["localhost"]
  ilm.enabled: true
  ilm.rollover_alias: "{beatname_lc}"
  ilm.pattern: "{now/d}-000001" <1>
----
<1> Date math is supported here. For more information, see
{ref}/indices-rollover-index.html#_using_date_math_with_the_rollover_api[Using
date math with the rollover API].

NOTE: If you modify the `rollover_alias` or `pattern` settings after loading the
index template, you must overwrite the template to apply the changes.
--

. Load the default policy into {es}. You can either use the `setup` command to
load the policy without modifying it, or modify the policy and load it manually.
+
--
To use the setup command, run:

["source","shell",subs="attributes"]
----
{beatname_lc} setup --ilm-policy
----

After loading the default policy, you can edit it in the *Index lifecycle policies*
UI in {kib}. For more information about working with the UI, see
{kibana-ref}/index-lifecycle-policies.html[Index lifecyle policies]. 

To modify the default policy before loading it, run +{beatname_lc} export
ilm-policy+ to print the policy to stdout. Modify the policy then use the
{ref}/ilm-put-lifecycle.html[Create lifecycle policy API] to load it into {es}. 

--

[float]
=== Advanced ILM settings

WARNING: We recommend that you avoid modifying these settings unless you know
what you're doing.

The default index lifecycle management settings work best for common use cases
that work with the automated alias setup described earlier. It is possible to
use a multiple write alias with dynamic index patterns, but this requires manual
set up. This section describes the configuration options you need to change. 

Let's assume you have the index pattern `customname-%{event.module}` where
`event.module` can have the values `system` and `apache`. First you must set up
a rollover index for `customname-system` and `customname-apache`. For details on
how to do this, see
{ref}/indices-rollover-index.html#_using_date_math_with_the_rollover_api[Rollover
Index]. 

Next, set the index pattern in the {es} output. For example:

["source","yaml",subs="attributes"]
----
output.elasticesarch.index: customname-%{event.module} <1>
----
<1> For this example to work, every event must contain `event.module`.

If you change the index name, you must also set the template name, template
pattern, rollover alias, and lifecycle name. The best way to set these is
through an {es} template. It's possible to disable the template loading in
{beatname_uc} and specify these settings in your own template. Or you can use
the following config options in {beatname_uc}:

[source,yaml]
----
setup.template.name: "customname"
setup.template.pattern: "customname-*"
setup.template.settings.index.lifecycle.rollover_alias: "customname"
setup.template.settings.index.lifecycle.name: "beats-default-policy"
----

IMPORTANT: If you set the options manually as shown in this example, do *not*
set `ilm.enabled`, or the settings specified in the configuration file will be
overwritten.

This configuration results in a managed index named something like
+customname-{localdate}-000001+ and the following index settings:

["source","shell"]
----
"aliases" : {
  "customname" : {
    "is_write_index" : true
  }
},
...
  "index" : {
    "lifecycle" : {
      "name" : "beats-default-policy",
      "rollover_alias" : "customname"
    },    
----


