Added several new modules (#11)

* Added several new modules

* Updated unit tests

* Updated README

Author: FragmentedPacket

.travis.yml

@@ -9,7 +9,7 @@ env:
   global:
     - COLLECTION_NAMESPACE: fragmentedpacket
     - COLLECTION_NAME: netbox_modules
-    - COLLECTION_VERSION: 0.0.9
+    - COLLECTION_VERSION: 0.1.0
 
 matrix:
   include:
@@ -42,7 +42,7 @@ matrix:
         - ansible-galaxy collection install $COLLECTION_NAMESPACE-$COLLECTION_NAME-$COLLECTION_VERSION.tar.gz -p /home/travis/.ansible/collections
 
       script:
-        - ansible-test units --python $PYTHON_VER -vvv
+        - ansible-test units --python $PYTHON_VER -v
         - black . --check
         - "sleep 60"
         - python tests/integration/netbox-deploy.py
@@ -77,7 +77,7 @@ matrix:
         - ansible-galaxy collection install $COLLECTION_NAMESPACE-$COLLECTION_NAME-$COLLECTION_VERSION.tar.gz -p /home/travis/.ansible/collections
 
       script:
-        - ansible-test units --python $PYTHON_VER -vv
+        - ansible-test units --python $PYTHON_VER -v
         - black . --check
         - "sleep 60"
         - python tests/integration/netbox-deploy.py

README.md

@@ -1,5 +1,5 @@
-[![Build Status](https://travis-ci.org/FragmentedPacket/netbox_modules.svg?branch=master)](https://travis-ci.org/FragmentedPacket/netbox_modules) 
-# Netbox modules for Ansible using Ansible Collections 
+[![Build Status](https://travis-ci.org/FragmentedPacket/netbox_modules.svg?branch=master)](https://travis-ci.org/FragmentedPacket/netbox_modules)
+# Netbox modules for Ansible using Ansible Collections
 
 **THIS IS A WIP DUE TO COLLECTIONS BEING IN TECH REVIEW CURRENTLY (Ansible 2.8) BUT WILL BE UPDATED AS NECESSARY AS COLLECTIONS MATURES**
 
@@ -27,11 +27,22 @@
 
 - netbox_device
 - netbox_device_interface
+- netbox_device_role
+- netbox_device_type
 - netbox_ip_address
+- netbox_ipam_role
+- netbox_manufacturer
+- netbox_platform
 - netbox_prefix
+- netbox_rack_group
+- netbox_rack_role
+- netbox_rack
 - netbox_site
 - netbox_tenant
 - netbox_tenant_group
+- netbox_vlan_group
+- netbox_vlan
+- netbox_vrf
 
 ## How to Use
 
@@ -68,15 +79,272 @@
 
 The structure of the Netbox modules attempts to follow the layout of the Netbox API by having a module_util for each application (`dcim, ipam, tenancy, etc`) that inherits from a base module (`NetboxModule - netbox_utils.py`) and then implements the specific endpoints within the correct application module.
 
-ex. Add logic for adding devices under netbox_dcim.py or ip addresses under netbox_ipam.py
+e.g. Add logic for adding devices under netbox_dcim.py or ip addresses under netbox_ipam.py
 
 In turn when creating the actual modules, we're just calling a single function and passing in the Ansible Module and the endpoint. This means all the logic is within the specific application's module_util module and a lot of the logic should be the same for most endpoints since it is a basic operation of using the desired state of the endpoint and then either making sure it exists, updating it if it does exist, or removing it. There may be some special logic for other endpoints, but it should be minimal.
 
 (Ansible Module) netbox_{{ endpoint }} -> (Module Util) netbox_{{ application }} -> (Module Util) netbox_utils
 
-### Testing
+These modules are built using the pynetbox Python library which allows you to interact with Netbox using objects. Most of this is abstracted away when creating more modules, but something to be aware of. The reasoning for using underscores within the endpoint names is so the endpoints work with pynetbox.
 
- 1. Please update `tests/unit/module_utils/test_netbox_base_class.py` if editing anything within the base class that needs to be tested.
- 2. Please add or update an existing play to test the new Netbox module for integration testing within `tests/integration/integration-tests.yml`
- 3. Run `black .` within the base directory for black formatting as it's required for tests to pass
- 4. Check necessary dependencies defined within `.travis.yml` for now if you're wanting to test locally
+An example of connecting to a Netbox instance and then choosing the application, endpoint, and operation:
+```python
+import pynetbox
+
+nb = pynetbox.api("http://localhost:32768", "0123456789abcdef0123456789abcdef01234567")
+
+# applications
+nb.circuits
+nb.dcim
+nb.extras
+nb.ipam
+nb.secrets
+nb.tenancy
+nb.virtualization
+
+# endpoints (small sample)
+nb.circuits.providers
+nb.dcim.devices
+nb.dcim.device_types
+nb.ipam.vrfs
+nb.ipam.ip_addresses
+nb.tenancy.tenant_groups
+
+# operations
+## Grabs a list of all endpoints
+nb.dcim.devices.**all**
+## Can pass a list of dicts to create multiple of the endpoints or just a dict to create a single endpoint
+nb.dcim.devices.**create**
+## Can filter to grab a name of the endpoint being filtered, not an object (Uses the same search criteria as the API)
+nb.dcim.devices.**filter**
+e.g. nb.dcim.devices.filter(name="test")
+## Will retrieve the actual object that can be manipulated (updated, etc.) (Uses the same search criteria as the API)
+nb.dcim.devices.**get**
+e.g. nb.dcim.devices.get(name="test")
+
+# Manipulate object after using .get
+## Now you can manipulate the object the same as a Python object
+device = nb.dcim.devices.get(name="test")
+device.description = "Test Description"
+## Patch operation (patches the data to the API)
+device.save()
+
+## If you were to just update the data in a fell swoop
+serial = {"serial": "FXS10001", "description": "Test Description"}
+## this operation will update the device and use the .save() method behind the scenes
+device.update(serial)
+```
+
+### Adding an Endpoint
+
+#### Updating Variables within Module Utils
+
+First thing is to setup several variables within **netbox_utils** and **netbox_application** module utils:
+
+Check the following variable to make sure the endpoint is within the correct application within **netbox_utils**:
+
+```python
+API_APPS_ENDPOINTS = dict(
+    circuits=[],
+    dcim=[
+        "devices",
+        "device_roles",
+        "device_types",
+        "interfaces",
+        "manufacturers",
+        "platforms",
+        "racks",
+        "rack_groups",
+        "rack_roles",
+        "regions",
+        "sites",
+    ],
+    extras=[],
+    ipam=["ip_addresses", "prefixes", "roles", "vlans", "vlan_groups", "vrfs"],
+    secrets=[],
+    tenancy=["tenants", "tenant_groups"],
+    virtualization=["clusters"],
+)
+```
+
+Create a new variable in the **netbox_application** module until that matches the endpoint with any spaces being converted to underscores and all lowercase:
+
+```python
+NB_DEVICE_TYPES = "device_types"
+```
+
+Add the endpoint to the **run** method of supported endpoints:
+
+```python
+class NetboxDcimModule(NetboxModule):
+    def __init__(self, module, endpoint):
+        super().__init__(module, endpoint)
+
+    def run(self):
+        """
+        This function should have all necessary code for endpoints within the application
+        to create/update/delete the endpoint objects
+        Supported endpoints:
+        - device_types
+```
+
+Add the endpoint to the **ENDPOINT_NAME_MAPPING** variable within the **netbox_utils** module util.
+
+```python
+ENDPOINT_NAME_MAPPING = {
+    "device_types": "device_type",
+}
+```
+
+Log into your Netbox instance and navigate to `/api/docs` and searching for the **POST** documents for the given endpoint you're looking to create.
+![POST Results](docs/media/postresults.PNG)
+The module should implement all available fields that are not the **id** or **readOnly** such as the **created, last_updated, device_count** in the example above.
+
+Add the endpoint to the **ALLOWED_QUERY_PARAMS** variable within the **netbox_utils** module util. This should be something unique for the endpoint and will be used within the **_build_query_params** method to dynamically build query params.
+
+```python
+ALLOWED_QUERY_PARAMS = {
+    "device_type": set(["slug"]),
+}
+```
+
+If the endpoint has a key that uses an **Array**, you will need to check the **_choices** of the application the endpoint is in and build those into **netbox_utils** module util.
+
+```python
+SUBDEVICE_ROLES = dict(parent=True, child=False)
+
+REQUIRED_ID_FIND = {
+    "device_types": [{"subdevice_role": SUBDEVICE_ROLES}],
+}
+# This is the method that uses the REQUIRED_ID_FIND variable (no change should be required within the method)
+def _change_choices_id(self, endpoint, data):
+    """Used to change data that is static and under _choices for the application.
+    e.g. DEVICE_STATUS
+    :returns data (dict): Returns the user defined data back with updated fields for _choices
+    :params endpoint (str): The endpoint that will be used for mapping to required _choices
+    :params data (dict): User defined data passed into the module
+    """
+    if REQUIRED_ID_FIND.get(endpoint):
+        required_choices = REQUIRED_ID_FIND[endpoint]
+        for choice in required_choices:
+            for key, value in choice.items():
+                if data.get(key):
+                    try:
+                        data[key] = value[data[key].lower()]
+                    except KeyError:
+                        self._handle_errors(
+                            msg="%s may not be a valid choice. If it is valid, please submit bug report."
+                            % (key)
+                        )
+
+    return data
+```
+
+If the key is something that pertains to a different endpoint such as **manufacturer** it will need to be added to a few variables within **netbox_utils**.
+
+```python
+CONVERT_TO_ID = dict(
+    manufacturer="manufacturers",
+)
+QUERY_TYPES = dict(
+    manufacturer="slug",
+)
+```
+
+If **slug** and **name** is required, we should leave **slug** out as an option within the module docs and generate it dynamically. Add the endpoint to **SLUG_REQUIRED** within **netbox_utils** module util.
+
+```python
+SLUG_REQUIRED = {
+    "device_roles",
+    "ipam_roles",
+    "rack_groups",
+    "rack_roles",
+    "roles",
+    "manufacturers",
+    "platforms",
+    "vlan_groups",
+}
+```
+
+Add code to the **netbox_application** module util to convert name to **slug**"
+
+```python
+if self.endpoint in SLUG_REQUIRED:
+    if not data.get("slug"):
+        data["slug"] = self._to_slug(name)
+```
+
+If either **role** or **group** are within the acceptable keys to POST to the endpoint, we should prefix it with the endpoint name. This is to prevent the code from trying to fetch an ID from the wrong endpoint.
+Add the new key to **CONVERT_KEYS** within **netbox_utils** module util.
+
+```python
+CONVERT_KEYS = {
+    "prefix_role": "role",
+    "rack_group": "group",
+    "rack_role": "role",
+    "tenant_group": "group",
+    "vlan_role": "role",
+    "vlan_group": "group",
+}
+
+# Adding the method that uses this code (no change should be required within the method)
+def _convert_identical_keys(self, data):
+    """
+    Used to change non-clashing keys for each module into identical keys that are required
+    to be passed to pynetbox
+    ex. rack_role back into role to pass to Netbox
+    Returns data
+    :params data (dict): Data dictionary after _find_ids method ran
+    """
+    for key in data:
+        if key in CONVERT_KEYS:
+            new_key = CONVERT_KEYS[key]
+            value = data.pop(key)
+            data[new_key] = value
+
+    return data
+```
+
+#### Creating **netbox_endpoint** Module
+
+Copying an existing module that has close to the same options is typically the path to least resistence and then updating portions of it to fit the new module.
+
+- Change the author: `Copyright: (c) 2018, Mikhail Yohman (@FragmentedPacket) <mikhail.yohman@gmail.com>`
+- Update the **DOCUMENTATION**/**EXAMPLES**/**RETURN** string with the necessary information
+  - Main things are module, descriptions, author, version and the sub options under data
+  - The **RETURN** should return the singluar of the endpoint name (done dynamically, but needs to be documented correctly)
+- Update the module_util, module, and endpoint variable for the endpoint
+
+  ```python
+  from ansible_collections.fragmentedpacket.netbox_modules.plugins.module_utils.netbox_dcim import (
+      NetboxDcimModule,
+      NB_DEVICE_ROLES,
+  )
+  ```
+
+- Update the **main()** as necessary:
+
+  ```python
+  # Add if name is required or change to match required fields
+  if not module.params["data"].get("name"):
+      module.fail_json(msg="missing name")
+  # Make sure the objects are endpoint name and the correct class and variable are being called for the endpoint
+  netbox_device_role = NetboxDcimModule(module, NB_DEVICE_ROLES)
+  netbox_device_role.run()
+  ```
+
+#### Testing
+
+- Please update `tests/unit/module_utils/test_netbox_base_class.py` if editing anything within the base class that needs to be tested. This will most likely be needed as there are a few unit tests that test the data of **ALLOWED_QUERY_PARAMS**, etc.
+
+  ```python
+  def test_normalize_data_returns_correct_data()
+  def test_find_app_returns_valid_app()
+  def test_change_choices_id()
+  def test_build_query_params_no_child()
+  def test_build_query_params_child()
+  ```
+
+- Please add or update an existing play to test the new Netbox module for integration testing within `tests/integration/integration-tests.yml`. Make sure to test creation, duplicate, update (if possible), and deletion along with any other conditions that may want to be tested.
+- Run `black .` within the base directory for black formatting as it's required for tests to pass
+- Check necessary dependencies defined within `.travis.yml` for now if you're wanting to test locally

docs/media/postresults.PNG

@@ -9,7 +9,7 @@ namespace: fragmentedpacket
 name: netbox_modules
 
 # The version of the collection. Must be compatible with semantic versioning
-version: 0.0.9
+version: 0.1.0
 
 # The path to the Markdown (.md) readme file. This path is relative to the root of the collection
 readme: README.md