=== S3 and SQS Setup
Enable bucket notification: any new object creation in S3 bucket will also
create a notification through SQS. Please see
https://docs.aws.amazon.com/AmazonS3/latest/dev/ways-to-add-notification-config-to-bucket.html#step1-create-sqs-queue-for-notification[create-sqs-queue-for-notification]
for more details.
1. In SQS, edit policy document to create a new policy.
2. In S3 bucket, enable and configure event notification.
3. In order to make sure the S3-SQS setup is ready, upload a file into the S3
bucket and check if SQS gets a message showing that a new object is created with
its name.

[float]
=== Manual Testing
1. Upload fake log files into the S3 bucket that has SQS notification enabled.
2. Check from SQS if there are N messages received.
3. Start filebeat with `./filebeat -e` and check Kibana if there are events reported
with messages from the example logs. Depends on the number of log lines in each
fake log file, check if the number of events match the number of log lines total
from all log files.
4. Check SQS if messages are deleted successfully.
5. Interrupt the s3 input process by killing filebeat during processing new S3 logs,
check if messages in SQS are in flight instead of deleted.

[float]
=== Run s3_test.go
Instead of manual testing, `s3_test.go` includes some integration tests that can
be used for validating s3 input. In order to run `s3_test.go`, an AWS environment
with S3-SQS setup is needed. Please see `S3 and SQS Setup` for more details on
how to set up the environment. In the test, it does a cleaning first to remove
all old messages from SQS queue. Then upload a sample log file, which stores in
`./ftest/sample1.txt`, into S3 bucket. Test function calls `input.Run()`
function to read the notification message from SQS and find the log file in S3
target bucket and get the log message. After validating the events, another round
of cleaning will be done for SQS to remove the message.

Some environment variables are needed for testing:

|===
| Environment Variable | Sample Value
| QUEUE_URL | https://sqs.us-west-1.amazonaws.com/1234567/test-s3-notification
| AWS_PROFILE_NAME | test-mb
| S3_BUCKET_NAME | test-s3
| S3_BUCKET_REGION | us-west-1
|===

[float]
=== Parallel Processing Test
A basic test was done with three Filebeats running in parallel pointing to the same
SQS queue in AWS. There were 1000 messages available in the queue and each message
notifies a new S3 log has been generated. These S3 logs are simple .txt files and
each contains 10 log lines. With three Filebeats, the messages were processed
evenly without duplicating or missing messages. Test result looks like:

|=======
| Filebeat #  | Total # of Events | Total # of log files
| 1 |  3350 |  335
| 2| 3350 |  335
| 3| 3300 |  330
|=======

Please see more details in https://github.com/elastic/beats/issues/13457 regarding
to the test.
