:plugin: s3
:type: input
:default_codec: plain

///////////////////////////////////////////
START - GENERATED VARIABLES, DO NOT EDIT!
///////////////////////////////////////////
:version: %VERSION%
:release_date: %RELEASE_DATE%
:changelog_url: %CHANGELOG_URL%
:include_path: ../../../../logstash/docs/include
///////////////////////////////////////////
END - GENERATED VARIABLES, DO NOT EDIT!
///////////////////////////////////////////

[id="plugins-{type}s-{plugin}"]

=== S3 input plugin

include::{include_path}/plugin_header.asciidoc[]

==== Description

Stream events from files from a S3 bucket.

Each line from each file generates an event.
Files ending in `.gz` are handled as gzip'ed files.

Files that are archived to AWS Glacier will be skipped.

[id="plugins-{type}s-{plugin}-options"]
==== S3 Input Configuration Options

This plugin supports the following configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.

[cols="<,<,<",options="header",]
|=======================================================================
|Setting |Input type|Required
| <<plugins-{type}s-{plugin}-access_key_id>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-additional_settings>> |<<hash,hash>>|No
| <<plugins-{type}s-{plugin}-aws_credentials_file>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-backup_add_prefix>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-backup_to_bucket>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-backup_to_dir>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-bucket>> |<<string,string>>|Yes
| <<plugins-{type}s-{plugin}-delete>> |<<boolean,boolean>>|No
| <<plugins-{type}s-{plugin}-endpoint>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-exclude_pattern>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-include_object_properties>> |<<boolean,boolean>>|No
| <<plugins-{type}s-{plugin}-interval>> |<<number,number>>|No
| <<plugins-{type}s-{plugin}-prefix>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-proxy_uri>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-region>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-role_arn>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-role_session_name>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-secret_access_key>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-session_token>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-sincedb_path>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-temporary_directory>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-watch_for_new_files>> |<<boolean,boolean>>|No
|=======================================================================

Also see <<plugins-{type}s-{plugin}-common-options>> for a list of options supported by all
input plugins.

&nbsp;

[id="plugins-{type}s-{plugin}-access_key_id"]
===== `access_key_id` 

  * Value type is <<string,string>>
  * There is no default value for this setting.

This plugin uses the AWS SDK and supports several ways to get credentials, which will be tried in this order:

1. Static configuration, using `access_key_id` and `secret_access_key` params in logstash plugin config
2. External credentials file specified by `aws_credentials_file`
3. Environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
4. Environment variables `AMAZON_ACCESS_KEY_ID` and `AMAZON_SECRET_ACCESS_KEY`
5. IAM Instance Profile (available when running inside EC2)

[id="plugins-{type}s-{plugin}-aws_credentials_file"]
===== `aws_credentials_file` 

  * Value type is <<string,string>>
  * There is no default value for this setting.

Path to YAML file containing a hash of AWS credentials.
This file will only be loaded if `access_key_id` and
`secret_access_key` aren't set. The contents of the
file should look like this:

[source,ruby]
----------------------------------
    :access_key_id: "12345"
    :secret_access_key: "54321"
----------------------------------


[id="plugins-{type}s-{plugin}-backup_add_prefix"]
===== `backup_add_prefix` 

  * Value type is <<string,string>>
  * Default value is `nil`

Append a prefix to the key (full path including file name in s3) after processing.
If backing up to another (or the same) bucket, this effectively lets you
choose a new 'folder' to place the files in

[id="plugins-{type}s-{plugin}-backup_to_bucket"]
===== `backup_to_bucket` 

  * Value type is <<string,string>>
  * Default value is `nil`

Name of a S3 bucket to backup processed files to.

[id="plugins-{type}s-{plugin}-backup_to_dir"]
===== `backup_to_dir` 

  * Value type is <<string,string>>
  * Default value is `nil`

Path of a local directory to backup processed files to.

[id="plugins-{type}s-{plugin}-bucket"]
===== `bucket` 

  * This is a required setting.
  * Value type is <<string,string>>
  * There is no default value for this setting.

The name of the S3 bucket.

[id="plugins-{type}s-{plugin}-delete"]
===== `delete` 

  * Value type is <<boolean,boolean>>
  * Default value is `false`

Whether to delete processed files from the original bucket.

[id="plugins-{type}s-{plugin}-endpoint"]
===== `endpoint`

  * Value type is <<string,string>>
  * There is no default value for this setting.

The endpoint to connect to. By default it is constructed using the value of `region`.
This is useful when connecting to S3 compatible services, but beware that these aren't
guaranteed to work correctly with the AWS SDK.

[id="plugins-{type}s-{plugin}-exclude_pattern"]
===== `exclude_pattern` 

  * Value type is <<string,string>>
  * Default value is `nil`

Ruby style regexp of keys to exclude from the bucket

[id="plugins-{type}s-{plugin}-additional_settings"]
===== `additional_settings`

  * Value type is <<hash,hash>>
  * Default value is `{}`

Key-value pairs of settings and corresponding values used to parametrize
the connection to s3. See full list in https://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html[the AWS SDK documentation]. Example:

[source,ruby]
    input {
      s3 {
        "access_key_id" => "1234"
        "secret_access_key" => "secret"
        "bucket" => "logstash-test"
        "additional_settings" => {
          "force_path_style" => true
          "follow_redirects" => false
        }
      }
    }

[id="plugins-{type}s-{plugin}-include_object_properties"]
===== `include_object_properties`

  * Value type is <<boolean,boolean>>
  * Default value is `false`

Whether or not to include the S3 object's properties (last_modified, content_type, metadata) into each Event at
`[@metadata][s3]`. Regardless of this setting, `[@metdata][s3][key]` will always be present.

[id="plugins-{type}s-{plugin}-interval"]
===== `interval` 

  * Value type is <<number,number>>
  * Default value is `60`

Interval to wait between to check the file list again after a run is finished.
Value is in seconds.

[id="plugins-{type}s-{plugin}-prefix"]
===== `prefix` 

  * Value type is <<string,string>>
  * Default value is `nil`

If specified, the prefix of filenames in the bucket must match (not a regexp)

[id="plugins-{type}s-{plugin}-proxy_uri"]
===== `proxy_uri` 

  * Value type is <<string,string>>
  * There is no default value for this setting.

URI to proxy server if required

[id="plugins-{type}s-{plugin}-region"]
===== `region` 

  * Value type is <<string,string>>
  * Default value is `"us-east-1"`

The AWS Region

[id="plugins-{type}s-{plugin}-role_arn"]
===== `role_arn`

  * Value type is <<string,string>>
  * There is no default value for this setting.

The AWS IAM Role to assume, if any.
This is used to generate temporary credentials, typically for cross-account access.
See the https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html[AssumeRole API documentation] for more information.

[id="plugins-{type}s-{plugin}-role_session_name"]
===== `role_session_name`

  * Value type is <<string,string>>
  * Default value is `"logstash"`

Session name to use when assuming an IAM role.

[id="plugins-{type}s-{plugin}-secret_access_key"]
===== `secret_access_key` 

  * Value type is <<string,string>>
  * There is no default value for this setting.

The AWS Secret Access Key

[id="plugins-{type}s-{plugin}-session_token"]
===== `session_token` 

  * Value type is <<string,string>>
  * There is no default value for this setting.

The AWS Session token for temporary credential

[id="plugins-{type}s-{plugin}-sincedb_path"]
===== `sincedb_path` 

  * Value type is <<string,string>>
  * Default value is `nil`

Where to write the since database (keeps track of the date
the last handled file was added to S3). The default will write
sincedb files to in the directory '{path.data}/plugins/inputs/s3/'

If specified, this setting must be a filename path and not just a directory.

[id="plugins-{type}s-{plugin}-temporary_directory"]
===== `temporary_directory` 

  * Value type is <<string,string>>
  * Default value is `"/tmp/logstash"`

Set the directory where logstash will store the tmp files before processing them.

[id="plugins-{type}s-{plugin}-watch_for_new_files"]
===== `watch_for_new_files`

  * Value type is <<boolean,boolean>>
  * Default value is `true`

Whether or not to watch for new files.
Disabling this option causes the input to close itself after processing the files from a single listing.

[id="plugins-{type}s-{plugin}-common-options"]
include::{include_path}/{type}.asciidoc[]

:default_codec!:
