Docs for metrics, FAQ, YARA matches, Lambda versions (#65)

Author: austinbyers

.gitignore

@@ -28,3 +28,6 @@ Thumbs.db
 .DS_Store
 *.swp
 terminal.glue
+
+# Documentation build
+docs/build/

README.md

@@ -0,0 +1,36 @@
+BinaryAlert: Serverless, Real-Time & Retroactive Malware Detection
+==================================================================
+.. image:: https://travis-ci.org/airbnb/binaryalert.svg?branch=master
+  :target: https://travis-ci.org/airbnb/binaryalert
+  :alt: Build Status
+
+.. image:: https://coveralls.io/repos/github/airbnb/binaryalert/badge.svg?branch=master
+  :target: https://coveralls.io/github/airbnb/binaryalert?branch=master
+  :alt: Coverage Status
+
+.. image:: https://readthedocs.org/projects/binaryalert/badge/?version=latest
+  :target: http://www.binaryalert.io/?badge=latest
+  :alt: Documentation Status
+
+|
+
+.. image:: docs/images/logo.png
+  :align: center
+  :scale: 75%
+  :alt: BinaryAlert Logo
+
+BinaryAlert is an open-source serverless AWS pipeline where any file uploaded to an S3 bucket is
+immediately scanned with a configurable set of `YARA <https://virustotal.github.io/yara/>`_ rules.
+An alert will fire as soon as any match is found, giving an incident response team the ability to
+quickly contain the threat before it spreads.
+
+Read the documentation at `binaryalert.io <https://binaryalert.io>`_!
+
+
+Links
+-----
+
+- `Announcement Post <https://medium.com/airbnb-engineering/binaryalert-real-time-serverless-malware-detection-ca44370c1b90>`_
+- `Documentation <https://binaryalert.io>`_
+- `Twitter <https://twitter.com/binaryalert_io>`_ (unofficial)
+- `Slack <https://binaryalert.herokuapp.com>`_ (unofficial)

README.rst

@@ -54,11 +54,15 @@ You can use these variables in your own rules to match or exclude certain filepa
   }
 
 
+.. _supported_yara_modules:
+
 Supported Modules
 -----------------
 BinaryAlert supports all of the default `YARA modules <http://yara.readthedocs.io/en/latest/modules.html>`_, including ELF, Math, Hash, and PE.
 
 
+.. _testing_yara_rules:
+
 Testing Your Rules
 ------------------
 The easiest way to test individual YARA rules is to `install YARA locally <http://yara.readthedocs.io/en/latest/gettingstarted.html#getting-started>`_. Note that you will need the ``-d`` flag to define external variables. For example, to test the ``evil_at_home`` rule above:

docs/source/adding-yara-rules.rst

@@ -15,6 +15,6 @@ Analysis Lifecycle
 2. Every file uploaded to the S3 bucket is immediately queued for analysis (using `S3 event notifications <http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html>`_).
 3. A dispatching Lambda function runs every minute, grouping files into batches and invoking up to dozens of analyzers in parallel.
 4. Each analyzer scans its files using a list of pre-compiled `YARA rules <adding-yara-rules.html>`_.
-5. YARA matches are saved to DynamoDB and an alert is sent to an SNS topic. You can subscribe to these alerts via `StreamAlert <https://streamalert.io>`_, email, or any other supported `SNS subscription <http://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html>`_.
+5. `YARA matches <yara-matches.html>`_ are saved to DynamoDB and an alert is sent to an SNS topic. You can subscribe to these alerts via `StreamAlert <https://streamalert.io>`_, email, or any other supported `SNS subscription <http://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html>`_.
 6. For retroactive analysis, a batching Lambda function enqueues the entire S3 bucket to be re-analyzed.
-7. Configurable CloudWatch alarms will trigger if any BinaryAlert component is behaving abnormally. This will notify a different SNS topic than the one used for YARA match alerts.
+7. Configurable :ref:`CloudWatch alarms <metric_alarms>` will trigger if any BinaryAlert component is behaving abnormally. This will notify a different SNS topic than the one used for YARA match alerts.

docs/source/architecture.rst

@@ -18,6 +18,22 @@ A ``deploy`` is equivalent to the following 4 operations executed in sequence:
 .. warning:: To ensure new YARA rules are applied ASAP, **every** ``deploy`` starts a batch analysis. If a batch analysis is already running or if you are not updating any YARA rules, you can just ``build`` and ``apply`` your changes.
 
 
+.. _lambda_versioning:
+
+Lambda Versions and Aliases
+---------------------------
+Each BinaryAlert Lambda function has a ``Production`` alias which points to the most recent version of that function. Every time a deploy changes one of the Lambda deployment packages, a new version is published and the ``Production`` alias is updated accordingly. For more information, see `AWS Lambda Function Versioning and Aliases <http://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html>`_.
+
+
+.. _add_sns_subscriptions:
+
+Add SNS Subscriptions
+---------------------
+BinaryAlert sends YARA match alerts to an `SNS <https://aws.amazon.com/sns/>`_ topic. In order to receive these alerts, you must manually `add a subscription <http://docs.aws.amazon.com/sns/latest/dg/SubscribeTopic.html>`_ to the generated ``NAME_PREFIX_binaryalert_yara_match_alerts`` topic. SNS supports a variety of subscription endpoints, including email, SMS, and other Lambda functions. Email/SMS subscriptions must be confirmed by the destination, which is why this step can't be automated with Terraform.
+
+For example, since `StreamAlert <https://streamalert.io>`_ supports `SNS datasources <https://streamalert.io/datasources.html#aws-sns>`_, you could use StreamAlert to forward the YARA match alert to PagerDuty, Slack, etc.
+
+
 Terraform State
 ---------------
 By default, Terraform will save the state of the infrastructure locally in ``terraform/terraform.tfstate``. If you are deploying BinaryAlert in an enterprise environment, we recommend configuring `Terraform remote state <https://www.terraform.io/docs/state/remote.html>`_. For example, you can store the Terraform state in a versioned `S3 bucket <https://www.terraform.io/docs/backends/types/s3.html>`_.
@@ -34,6 +50,8 @@ We recommend using the ``manage.py`` wrapper script for most BinaryAlert managem
   $ terraform show  # Print the current state of the infrastructure
 
 
+.. _terraform_destroy:
+
 Terraform Destroy
 .................
 To teardown all of the BinaryAlert infrastructure:
@@ -43,4 +61,4 @@ To teardown all of the BinaryAlert infrastructure:
   $ cd terraform/
   $ terraform destroy
 
-.. note:: By default, S3 objects will not be deleted by ``terraform destroy``. To do so, you have to enable the ``force_destroy`` option in the ``terraform/terraform.tfvars`` configuration file.
+.. note:: By default, S3 objects will not be deleted by ``terraform destroy``. To do so, you must first enable the ``force_destroy`` option in the ``terraform/terraform.tfvars`` configuration file and ``apply`` the change.

docs/source/deploying.rst

@@ -1,11 +1,11 @@
 Getting Started
 ===============
-All you need is a computer and an AWS account to get BinaryAlert up and running in just a few minutes!
+All you need is an AWS account to get BinaryAlert up and running in just a few minutes!
 
 
 Install Dependencies
 --------------------
-BinaryAlert can be deployed from any MacOS/Linux environment (and likely Windows as well).
+BinaryAlert can be deployed from any MacOS/Linux environment (and likely Windows as well, though we haven't tried).
 
 1. Install `Python 3.6 <https://www.python.org/downloads/release/python-362>`_:
 
@@ -58,7 +58,7 @@ Download BinaryAlert
 
 .. code-block:: bash
 
-  $ git clone --branch 0.10 --depth 1 https://github.com/airbnb/binaryalert
+  $ git clone --branch v1.0.0 --depth 1 https://github.com/airbnb/binaryalert
 
 2. Create and activate a virtual environment:
 
@@ -106,3 +106,5 @@ Deploy!
 .. code-block:: bash
 
   $ ./manage.py live_test
+
+.. note:: You must :ref:`add an SNS subscription <add_sns_subscriptions>` in order to receive YARA match alerts.

docs/source/getting-started.rst

@@ -42,3 +42,6 @@ Table of Contents
    adding-yara-rules
    deploying
    uploading-files
+   yara-matches
+   metrics-and-monitoring
+   troubleshooting-faq

docs/source/index.rst

@@ -0,0 +1,55 @@
+Metrics and Monitoring
+======================
+BinaryAlert automatically generates logs, custom metrics, alarms, and a dashboard to help you visualize its performance. These are all part of the `AWS CloudWatch <https://aws.amazon.com/cloudwatch/>`_ service.
+
+
+.. _cloudwatch_logs:
+
+Logging
+-------
+Each BinaryAlert Lambda function logs information about its execution; logs are saved for 60 days (by default) and are accessible from AWS CloudWatch.
+
+
+Custom Metrics
+--------------
+In addition to the wide array of `metrics provided automatically AWS <http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CW_Support_For_AWS.html>`_, BinaryAlert publishes the following custom metrics in the ``BinaryAlert`` namespace:
+
+====================  ============  =============================================
+**Metric Name**       **Unit**      **Description**
+--------------------  ------------  ---------------------------------------------
+AnalyzedBinaries      Count         Number of binaries analyzed
+BatchEnqueueFailures  Count         Number of SQS messages that failed to enqueue
+MatchedBinaries       Count         Number of binaries which matched a YARA rule
+S3DownloadLatency     Milliseconds  Time to download binaries from S3
+YaraRules             Count         Number of compiled YARA rules
+====================  ============  =============================================
+
+
+.. _metric_alarms:
+
+Metric Alarms
+-------------
+BinaryAlert creates metric alarms which trigger if BinaryAlert behavior is abnormal. Similar to :ref:`adding SNS subscriptions for YARA match alerts <add_sns_subscriptions>`, you will need to configure subscriptions for the ``NAME_PREFIX_binaryalert_metric_alarms`` SNS topic if you want to be notified about metric alarms.
+
+Alarms can be configured from the ``terraform/terraform.tfvars`` configuration file, or else by directly modifying ``terraform/cloudwatch_metric_alarm.tf``. The alarm defaults are as follows:
+
+=============  =============================  ======================================
+**Namespace**  **Metric Name**                **Alarm Condition**
+-------------  -----------------------------  --------------------------------------
+AWS/DynamoDB   ThrottledRequests              > 0
+AWS/Lambda     Errors                         > threshold (for each Lambda function)
+AWS/Lambda     Throttles                      > 0 (for each Lambda function)
+AWS/SQS        ApproximateAgeOfOldestMessage  >= 30 minutes
+BinaryAlert    AnalyzedBinaries               == 0 for an hour
+BinaryAlert    BatchEnqueueFailures           > 0
+BinaryAlert    YaraRules                      < 5
+=============  =============================  ======================================
+
+The description for each alarm includes context and troubleshooting information.
+
+
+.. _cloudwatch_dashboard:
+
+Dashboard
+---------
+The aforementioned metrics (and many others) are aggregated into a single BinaryAlert dashboard at the following URL (substitute your region for ``us-east-1``): `https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#dashboards:name=BinaryAlert <https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#dashboards:name=BinaryAlert>`_.