Add documentation for architecture, deploying, and uploading files (#64)

Author: austinbyers

docs/source/adding-yara-rules.rst

@@ -28,14 +28,19 @@ Write Your Own Rules
 --------------------
 You can add your own ``.yar`` or ``.yara`` files anywhere in the ``rules/`` directory tree. Refer to the `writing YARA rules <http://yara.readthedocs.io/en/latest/writingrules.html>`_ documentation for guidance and examples. Note that when BinaryAlert finds a file which matches a YARA rule, the rule name, `metadata <http://yara.readthedocs.io/en/latest/writingrules.html#metadata>`_, `tags <http://yara.readthedocs.io/en/latest/writingrules.html#rule-tags>`_, and matched `string <http://yara.readthedocs.io/en/latest/writingrules.html#strings>`_ names will be included in the alert for your convenience.
 
+
+.. _external-variables:
+
+External Variables
+------------------
 In order to support the rule repositories listed above, BinaryAlert provides the following `external variables <http://yara.readthedocs.io/en/latest/writingrules.html#external-variables>`_:
 
 * ``extension`` - File extension (".docx", ".exe", ".pdf", etc)
 * ``filename`` - File basename ("file.exe")
 * ``filepath`` - Full file path ("/path/to/file.exe")
 * ``filetype`` - Uppercase ``extension`` without leading period ("DOCX", "EXE", "PDF"), etc
 
-You can use these variables in your own rules to match or exclude certain filepaths. For example, this is a YARA rule which matches only files containing the string "evil" in the ``/home/`` directory:
+You can use these variables in your own rules to match or exclude certain filepaths. (Note that the variables will default to empty strings if they are not available.) For example, this is a YARA rule which matches only files containing the string "evil" in the ``/home/`` directory:
 
 .. code-block:: none
 

docs/source/architecture.rst

@@ -0,0 +1,20 @@
+Architecture
+============
+BinaryAlert utilizes a `serverless <https://aws.amazon.com/serverless/>`_ architecture which is low-cost and easy to scale and maintain. While it's helpful to understand how BinaryAlert works, keep in mind that Terraform manages all of these components so you don't have to!
+
+.. image:: ../images/architecture.png
+  :align: center
+  :scale: 80%
+  :alt: BinaryAlert Architecture
+
+
+Analysis Lifecycle
+------------------
+
+1. The organization collects files and `delivers them <uploading-files.html>`_ to their BinaryAlert S3 bucket. Files of interest could include executable binaries, email attachments, documents, etc.
+2. Every file uploaded to the S3 bucket is immediately queued for analysis (using `S3 event notifications <http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html>`_).
+3. A dispatching Lambda function runs every minute, grouping files into batches and invoking up to dozens of analyzers in parallel.
+4. Each analyzer scans its files using a list of pre-compiled `YARA rules <adding-yara-rules.html>`_.
+5. YARA matches are saved to DynamoDB and an alert is sent to an SNS topic. You can subscribe to these alerts via `StreamAlert <https://streamalert.io>`_, email, or any other supported `SNS subscription <http://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html>`_.
+6. For retroactive analysis, a batching Lambda function enqueues the entire S3 bucket to be re-analyzed.
+7. Configurable CloudWatch alarms will trigger if any BinaryAlert component is behaving abnormally. This will notify a different SNS topic than the one used for YARA match alerts.

docs/source/deploying.rst

@@ -0,0 +1,46 @@
+Deploying
+=========
+After you've `setup your environment <getting-started.html>`_, deploying BinaryAlert is as easy as:
+
+.. code-block:: bash
+
+  $ ./manage.py deploy
+
+A ``deploy`` is equivalent to the following 4 operations executed in sequence:
+
+.. code-block:: bash
+
+  $ ./manage.py unit_test    # Unit tests ensure YARA rules compile correctly
+  $ ./manage.py build        # Build the Lambda ".zip" deployment packages
+  $ ./manage.py apply        # Runs "terraform apply" to update the infrastructure
+  $ ./manage.py analyze_all  # Starts a batch analysis of the entire S3 bucket
+
+.. warning:: To ensure new YARA rules are applied ASAP, **every** ``deploy`` starts a batch analysis. If a batch analysis is already running or if you are not updating any YARA rules, you can just ``build`` and ``apply`` your changes.
+
+
+Terraform State
+---------------
+By default, Terraform will save the state of the infrastructure locally in ``terraform/terraform.tfstate``. If you are deploying BinaryAlert in an enterprise environment, we recommend configuring `Terraform remote state <https://www.terraform.io/docs/state/remote.html>`_. For example, you can store the Terraform state in a versioned `S3 bucket <https://www.terraform.io/docs/backends/types/s3.html>`_.
+
+
+Terraform Commands
+------------------
+We recommend using the ``manage.py`` wrapper script for most BinaryAlert management because it provides additional validation. However, you can run ``terraform`` commands directly from the ``terraform/`` directory. Examples:
+
+.. code-block:: bash
+
+  $ cd terraform/
+  $ terraform plan  # Show pending changes
+  $ terraform show  # Print the current state of the infrastructure
+
+
+Terraform Destroy
+.................
+To teardown all of the BinaryAlert infrastructure:
+
+.. code-block:: bash
+
+  $ cd terraform/
+  $ terraform destroy
+
+.. note:: By default, S3 objects will not be deleted by ``terraform destroy``. To do so, you have to enable the ``force_destroy`` option in the ``terraform/terraform.tfvars`` configuration file.

docs/source/index.rst

@@ -3,7 +3,8 @@ BinaryAlert
 
 .. image:: ../images/logo.png
   :align: center
-  :alt: BinaryAlert
+  :scale: 75%
+  :alt: BinaryAlert Logo
 
 
 BinaryAlert is a serverless, real-time framework for detecting malicious files. BinaryAlert can efficiently analyze millions of files a day with a configurable set of `YARA <http://virustotal.github.io/yara/>`_ rules and will trigger an alert as soon as anything malicious is discovered! Organizations can deploy BinaryAlert to their private AWS account in a matter of minutes, allowing them to analyze internal files and documents within the confines of their own environment.
@@ -27,14 +28,17 @@ Resources
 
 * `GitHub Repo <https://github.com/airbnb/binaryalert>`_
 * `Blog Post <https://medium.com/airbnb-engineering/binaryalert-real-time-serverless-malware-detection-ca44370c1b90>`_
-* `Slack <https://binaryalert.herokuapp.com/>`_
+* `Slack <https://binaryalert.herokuapp.com/>`_ (unofficial)
 * `Twitter <https://twitter.com/binaryalert_io>`_ (unofficial)
 
 
 Table of Contents
 =================
 .. toctree::
-   :maxdepth: 2
+   :maxdepth: 3
 
    getting-started
+   architecture
    adding-yara-rules
+   deploying
+   uploading-files

docs/source/uploading-files.rst

@@ -0,0 +1,85 @@
+Uploading Files
+===============
+To upload files for analysis, you need only upload them to the BinaryAlert S3 bucket. The S3 bucket name is of the form
+
+.. code-block:: none
+
+  YOUR.NAME.PREFIX.binaryalert-binaries.REGION
+
+When uploading to S3, any object metadata you set will be included in all match alerts. In addition, if there is a ``filepath`` metadata key, BinaryAlert will make the filepath :ref:`external-variables` available to the YARA rules.
+
+Uploaded files are persisted indefinitely so that BinaryAlert can retroactively analyze all files with every rule update. The S3 bucket has both `access logging <http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html>`_ and `object versioning <http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectVersioning.html>`_ enabled.
+
+
+CarbonBlack Downloader
+----------------------
+If you use CarbonBlack Enterprise Response, you can enable BinaryAlert's optional downloader Lambda function. The downloader copies files from CarbonBlack into BinaryAlert's S3 bucket (including the appropriate metadata). To enable it:
+
+.. code-block:: none
+
+  $ ./manage.py configure
+  AWS Region (us-east-1):
+  Unique name prefix, e.g. "company_team": your_unique_prefix
+  Enable the CarbonBlack downloader? (no): yes
+  CarbonBlack URL: https://your.carbonblack.url
+  CarbonBlack API token (only needs binary read access):
+
+.. warning:: The API token only needs access to read binaries; do not use a token with admin privileges, do not allow other users to share the same token, and be sure to regularly rotate the token.
+
+.. note:: The API token will not be shown on screen and BinaryAlert will create a new KMS key to encrypt the credentials before saving them to the ``terraform.tfvars`` configuration file. The downloader (and no other component) is authorized to decrypt the credentials with the generated key.
+
+Binaries downloaded from CarbonBlack are saved to the BinaryAlert S3 bucket with the key ``carbonblack/MD5`` and with the following metadata:
+
+.. code-block:: python
+
+  [
+      'carbon_black_group',
+      'carbon_black_host_count',
+      'carbon_black_last_seen',
+      'carbon_black_md5',
+      'carbon_black_os_type',
+      'carbon_black_virustotal_score',
+      'carbon_black_webui_link',
+      'filepath'  # from the "observed_filenames" CarbonBlack metadata
+  ]
+
+Once the downloader is enabled, you can either copy everything from CarbonBlack in one go, or you can `deploy <deploying.rst>`_ the downloader components and setup real-time invocations for every new binary.
+
+
+Copy All Files
+..............
+If you want to run a one-time job to copy every file from CarbonBlack into BinaryAlert:
+
+.. code-block:: bash
+
+  $ ./manage.py cb_copy_all
+
+This runs *locally*, using multiple threads to enumerate the files in CarbonBlack and copy them over to BinaryAlert. The downloader *code* is used, but there are no Lambda invocations. This means you can copy all of the files from CarbonBlack without actually deploying the downloader components.
+
+
+Real-Time Invocations
+.....................
+To ensure real-time file analysis, we recommend invoking the downloader every time CarbonBlack logs a ``binarystore.file.added`` event. If you use `StreamAlert <https://streamalert.io/>`_ to process CarbonBlack logs, the following `rule <https://streamalert.io/rules.html>`_ will invoke the BinaryAlert downloader for every new binary (assuming BinaryAlert is a properly configured Lambda `output <https://streamalert.io/outputs.html>`_):
+
+.. code-block:: python
+
+  @rule(logs=['carbonblack:binarystore.file.added'],
+        matchers=[],
+        outputs=['aws-lambda:binaryalert'])
+  def cb_binarystore_file_added(rec):
+      """
+      description: CarbonBlack found a new binary: forward to BinaryAlert for YARA analysis.
+      """
+      return True
+
+If you don't use StreamAlert, you can invoke the downloader yourself:
+
+.. code-block:: python
+
+  import boto3, json
+  boto3.client('lambda').invoke(
+      FunctionName='your_prefix_binaryalert_downloader',
+      InvocationType='Event',  # Asynchronous invocation
+      Qualifier='Production',  # Invoke production alias
+      Payload=json.dumps({'md5': 'FILE_MD5'}).encode('utf-8')
+  )

terraform/cloudwatch_dashboard.tf

@@ -95,11 +95,11 @@ EOF
     "annotations": {
       "horizontal": [
         {
-          "label": "Maximum Age",
+          "label": "Max",
           "value": ${aws_sqs_queue.s3_object_queue.message_retention_seconds}
         },
         {
-          "label": "High Age Alarm",
+          "label": "Alarm",
           "value": ${aws_cloudwatch_metric_alarm.sqs_age.threshold}
         }
       ]
@@ -163,7 +163,7 @@ EOF
     "annotations": {
       "horizontal": [
         {
-          "label": "Maximum",
+          "label": "Max",
           "value": 300000
         }
       ]