Logback RollingPolicy with S3 upload

logback-s3 is a Logback RollingPolicy that automaitcally uploads rolled log files to S3. As S3FixedWindowRollingPolicy extends FixedWindowRollingPolicy, it works exactly same as FixedWindowRollingPolicy apart from uploading log files to S3.

logback-s3 is implemented based on Logpic (https://github.com/mweagle/Logpig) but simplified for my needs.

Configuration logback.xml

If awsAccessKey or awsSecretKey are not set, Logback will create a AmazonS3Client with no parameters to look for credentials in the following places:

1. In the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_KEY.
2. In the Java system properties aws.accessKeyId and aws.secretKey.
3. From the Instance Metadata Service (IMDS). The IMDS is what provides credentials using the IAM role contained in the EC2 instance profile.

Override compressedFileNameSufix if you you want to add a sufix to the compressed file name

Example logback.xml

An example logback.xml that uses S3FixedWindowRollingPolicy with RollingFileAppender.

<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
  <file>/var/log/myapp.log</file>
  <encoder>
    <pattern>%d\t%thread\t%level\t%logger\t%msg%n</pattern>
  </encoder>

  <!--
    Policy to upload a log file into S3 on log rolling or JVM exit.
    * On each log rolling, a rolled log file is created locally and uploaded to S3
    * When <rollingOnExit> is true, log rolling occurs on JVM exit and a rolled log is uploaded (default)
    * When <rollingOnExit> is false, the active log file is uploaded as it is
  -->

  <rollingPolicy class="ch.qos.logback.core.rolling.S3FixedWindowRollingPolicy">
    <fileNamePattern>/var/log/myapp.log.%i.gz</fileNamePattern>
        <awsAccessKey>accesskey</awsAccessKey>
        <awsSecretKey>secretkey</awsSecretKey>
        <s3BucketName>com.mybucket</s3BucketName>
        <s3FolderName>log</s3FolderName>
        <rollingOnExit>true</rollingOnExit>
    </rollingPolicy>

    <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
        <maxFileSize>10MB</maxFileSize>
    </triggeringPolicy>
</appender>

By suppressing the log rolling as follows, the appender works exactly same as FileAppender plus S3 upload.

<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">

  ...

  <rollingPolicy class="ch.qos.logback.core.rolling.S3FixedWindowRollingPolicy">
    <!-- this file name won't be used since log rolling is suppressed -->
    <fileNamePattern>/var/log/myapp.log.%i.gz</fileNamePattern>

        ...

        <rollingOnExit>false</rollingOnExit>

        <!--
            Suppress rolling.
            Note that rollingOnExit = true does not work with this config.
         -->
        <maxIndex>-1</maxIndex>
        <minIndex>-1</minIndex>
    </rollingPolicy>

    ...

</appender>

It is a good idea to create an IAM user only allowed to upload S3 object to a specific S3 bucket. It improves the control and reduces the risk of unauthorized access to your S3 bucket.

The following is an example IAM policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:PutObject"
      ],
      "Sid": "Stmt1378251801000",
      "Resource": [
        "arn:aws:s3:::com.mybucket/log/*"
      ],
      "Effect": "Allow"
    }
  ]
}