Skip to content

Commit 5122fb1

Browse files
jeffkalaSusan HooksDavid CatesDav-CJeff Kala
authored
Prep 4.0.0b0 for a tagged release to pypi (#182)
* add a more advanced jdiff jpath query to ios for ni interfaces * updates for parsing * bump version * update network importer * add platform none check * fix formatting * add conditional for no platform * add conditional for no platform * add conditional for no platform * add unsupported check * add readtimeout * add failed task UI logging * add junos support to do * add logging for failed reason * add junos support to do * update junos, add json load to linterfaces * update junos, add json load to linterfaces * update ios to use tojson filter * update ios to use tojson filter * formatting, update function name and doc strings * use j2 sandbox and nb core render function * poetry lock and black * black * fix juniper_junos.yml * juniper_junos.yml formatting * update ssot network adapter, models and mock data * ios NI initial working * fix do for junos * add csv support wip * update csv import feature * update csv support * update csv support * fix dup ip address extraction * fix mgmt_interface from list to string * update csv support * add orm caching method to onboarding adapter * update csv support * remove print statement * send interfaces in ios as json * update csv support * bump version * update NI * remove tag filter option * remove tag filter * decouple nornir play from job to remove enqueue job option * clean up via black * update csv import logging * fix bug in _process_csv_data * remove command getter job * update NI * updates for description and enabled * update mock data * formatting * black * pylint * update logging message * flake8 * update CSV support * update lock file * update lock and toml files * update logging around existing devies * change command mapper formatting and DO for junos * fix junos * remove prefix_length from OnboardingDeviceModel * add data check for fields returned by devices * enhance junos mgmt int and mask captures * enhance junos mgmt int and mask captures * fixes for most jnos * rework yaml format and processor formatter * fixed nxos serial * improve logging and calls to secrets providers * update onboarding adapter * set defaut platform value to None when loading devices in onboarding adapter * yamllint, ruff * add more performant lookup to mgmt interface cisco ios * bump version, fix bug with DO adapter * update DO adapters, models and job * bump version * formatting changes * changed ip address to list * fix yamllint, pylint and juniper yaml jpaths * updates to NI formatting * fix junos mgmt intrface post processor * fix junos * Update juniper_junos.yml * Update juniper_junos.yml * update ni add nxos * update vlans * update NI adapters and models, add untagged vlan model * bump * linting * update lock file * updates for multiple devices * update mac address conversion * add cache for primary ips to NI adapter * update sync complete * update sync complete * update sync complete * add sync complete doc string * updated interface mapping * bump * black * pylint * add error handling * bump version * black * yamlint * updates for vrf * revamps and cleanups * update list conversion * more refactors and cleanups * fix custom filters development config * remove jinja2 fitler registration in favor of native support with decorator * linters, formatters, stuff * fix platform parsing info * fix return indent level * fix return indent level * add default dict * more cleanups remove transform function * more refactors and few cleanups * add vrf to ssot sync, fix vlan bug, update logging * remove unnecessary logging * updates for VRF support * removed exception for vrf without rd * first DO for WLC * added 802.1q support for ios * fix / add needed info for wlc * updates for vlans ios * adjust vrf adapter to load an RD of "None" if not present * update vrf model * update vrf model for NI * add schema and some more new filters and cleanups * update for handling * update * vlans add * first pass at adding docs for new ssot features * more doc adds * more doc adds * vlans for nxos working * linting * fix example dict keys * remove RD from vrf sync * update mock data * removed rd from vrf * add wlc mapper and support for do * update jinja_filters.py, lock and bump version * fix link in docs * update mkdocs * removed extra rd config * cleanup * Jkala prep beta (#174) * rename jobs throughout codebase * rename jobs throughout docs * few more updates to original definition * more rename changes with black * bump to beta version * fix lock file inconsistances * initial round of fixes for mapping etc. * change use_textfsm key to parser and update interworks phase1 * change use_textfsm key to parser and update interworks phase2 * interface mapping comments * fix dedup function to work with dict or list of commands * fix support of dict or list def of commands * use mapping netmiko specifically * mock data * add platform to NI device filters * updates to vlan and vrf * update dependencies, pre-merge linting, black * update lock * update nautobot version in github ci * update role filter params * fix error messages, add defaults * New framework and cleanups. (#178) * prototype working first commit * fix a few device onboarding issues * more updates to framework * getting nxos at least working with sync network data * finish renames and cleanup of old functions * logic based on vlans and vrfs toggle * sync net data for arista * testing * another round of fixes * handle main task failures in processor * handle main task failures in processor * handle main task failures in processor * update for wlc wlc_ssh oddities * wlc updates and upgrade ntc-templates to master branch * temp juniper * temp juniper * temp juniper * add lag and vrf to ios and xe more junos fixes * add lag and vrf to ios and xe more junos fixes * add new filter * fix lags for ios and xe * fix lags for ios and xe * fix lags for ios and xe * fix lags for ios and xe * fix logic for dot1q * sync ios and xe mappers * sync ios and xe mappers * more updates * more updates * working on interface syncs * ios and xe fix vrf, lags for xes with and without switchport command * junos work * junos work * junos work * junos work * junos work * junos work * junos work * junos work * junos work * more junos fixes * junos work * junos work * fix junos unlimited mtu issue * junos fix vrf jpath * junos fix vrf jpath and command * junos fix vrf jpath and command * fix lag parsing to only parent physical interface * junos fix vrf jpath and command * fix arista lags and vrfs * remove prints and add a specific logger for ETL * update to beta2 * linting and formatting cleanups * linting and formatting cleanups round 2 * fix lock file * revert change to dockerfile --------- Co-authored-by: Jeff Kala <jeff.kala@chevron.com> * few fixes for parsing issues when result is empty list * update lag constants for nxos * hide vlans sync option * fix vrf for ios and xe * add default of empty list * wlc fixes * wlc fixes * more juniper work * more juniper work * changed job description * fix junos ip address * fix bug in network data models * yamlint, black * update version and lock * add schema validation * clean up logging * add date tracking and additional error handling (#180) * add date tracking to sync * update logging and error handling * remove unnecessary line * black . * update lock * udpate comment --------- Co-authored-by: David Cates <david.cates@networktocode.com> * fix commit * add unittest v1 * remove merge conflict additions in processor, bandit cleanups * pylint disables and formatting * update import exception * add debug for ntc-template parsed result * clean up processor logging * add test for SyncDevicesNetworkAdapter * remove mock data file * add more test * first sync network data test and run formatting * update custom field * prep 4.0 beta0 release and get all test passing * fix sorting on test * fix sorting on test * add troubleshooting processor and hidden job * add hidden job with json loads for none parsers * finalize new unittest for sync devices --------- Co-authored-by: Susan Hooks <susan.hooks@chevron.com> Co-authored-by: David Cates <david.cates@networktocode.com> Co-authored-by: David Cates <57967713+Dav-C@users.noreply.github.com> Co-authored-by: Jeff Kala <jeff.kala@chevron.com>
1 parent ae6896b commit 5122fb1

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

45 files changed

+29581
-2405
lines changed

‎.gitignore

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -297,7 +297,7 @@ fabric.properties
297297

298298
# Rando
299299
creds.env
300-
development/*.txt
300+
development/
301301

302302
# Invoke overrides
303303
invoke.yml

‎development/docker-compose.base.yml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,6 @@ x-nautobot-base: &nautobot-base
1313
- "creds.env"
1414
tty: true
1515

16-
version: "3.8"
1716
services:
1817
nautobot:
1918
depends_on:

‎development/docker-compose.dev.yml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,6 @@
33
# any override will need to include these volumes to use them.
44
# see: https://github.com/docker/compose/issues/3729
55
---
6-
version: "3.8"
76
services:
87
nautobot:
98
command: "nautobot-server runserver 0.0.0.0:8080"

‎development/docker-compose.mysql.yml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,4 @@
11
---
2-
version: "3.8"
3-
42
services:
53
nautobot:
64
environment:

‎development/docker-compose.postgres.yml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,4 @@
11
---
2-
version: "3.8"
3-
42
services:
53
nautobot:
64
environment:

‎development/docker-compose.redis.yml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
---
2-
version: "3.8"
32
services:
43
redis:
54
image: "redis:6-alpine"

‎development/nautobot_config.py

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -127,11 +127,7 @@
127127
#
128128

129129
# Enable installed Apps. Add the name of each App to the list.
130-
PLUGINS = [
131-
"nautobot_device_onboarding",
132-
"nautobot_ssot",
133-
"nautobot_plugin_nornir",
134-
]
130+
PLUGINS = ["nautobot_device_onboarding", "nautobot_ssot", "nautobot_plugin_nornir"]
135131

136132
# Apps configuration settings. These settings are used by various Apps that the user may have installed.
137133
# Each key in the dictionary is the name of an installed App and its value is a dictionary of settings.

‎nautobot_device_onboarding/choices.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
"""Choices used througout the app."""
2+
3+
SSOT_JOB_TO_COMMAND_CHOICE = (
4+
("sync_devices", "Sync Devices"),
5+
("sync_network_data", "Sync Network Data"),
6+
("both", "Both"),
7+
)

‎nautobot_device_onboarding/diffsync/adapters/sync_network_data_adapters.py

Lines changed: 142 additions & 110 deletions
Large diffs are not rendered by default.

‎nautobot_device_onboarding/diffsync/models/sync_network_data_models.py

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,19 @@
22

33
from typing import List, Optional
44

5+
try:
6+
from typing import Annotated # Python>=3.9
7+
except ImportError:
8+
from typing_extensions import Annotated # Python<3.9
9+
510
from diffsync import DiffSync, DiffSyncModel
611
from diffsync import exceptions as diffsync_exceptions
712
from django.core.exceptions import MultipleObjectsReturned, ObjectDoesNotExist, ValidationError
813
from nautobot.dcim.choices import InterfaceTypeChoices
914
from nautobot.dcim.models import Device, Interface, Location
1015
from nautobot.extras.models import Status
1116
from nautobot.ipam.models import VLAN, VRF, IPAddress, IPAddressToInterface
12-
from nautobot_ssot.contrib import NautobotModel
17+
from nautobot_ssot.contrib import CustomFieldAnnotation, NautobotModel
1318

1419
from nautobot_device_onboarding.utils import diffsync_utils
1520

@@ -48,11 +53,16 @@ class SyncNetworkDataDevice(FilteredNautobotModel):
4853
"name",
4954
"serial",
5055
)
56+
_attributes = ("last_network_data_sync",)
5157
_children = {"interface": "interfaces"}
5258

5359
name: str
5460
serial: str
5561

62+
last_network_data_sync: Annotated[
63+
Optional[str], CustomFieldAnnotation(key="last_network_data_sync", name="last_network_data_sync")
64+
]
65+
5666
interfaces: List["SyncNetworkDataInterface"] = []
5767

5868
@classmethod

‎nautobot_device_onboarding/jobs.py

Lines changed: 105 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,15 +7,19 @@
77

88
from diffsync.enum import DiffSyncFlags
99
from django.conf import settings
10+
from django.contrib.contenttypes.models import ContentType
1011
from django.core.exceptions import ObjectDoesNotExist, ValidationError
1112
from django.forms import HiddenInput
12-
from nautobot.apps.jobs import BooleanVar, FileVar, IntegerVar, Job, MultiObjectVar, ObjectVar, StringVar
13+
from nornir import InitNornir
14+
from nornir.core.plugins.inventory import InventoryPluginRegister
15+
from nautobot.apps.jobs import BooleanVar, FileVar, IntegerVar, Job, MultiObjectVar, ObjectVar, StringVar, ChoiceVar
1316
from nautobot.core.celery import register_jobs
1417
from nautobot.dcim.models import Device, DeviceType, Location, Platform
15-
from nautobot.extras.choices import SecretsGroupAccessTypeChoices, SecretsGroupSecretTypeChoices
16-
from nautobot.extras.models import Role, SecretsGroup, SecretsGroupAssociation, Status
18+
from nautobot.extras.choices import CustomFieldTypeChoices, SecretsGroupAccessTypeChoices, SecretsGroupSecretTypeChoices
19+
from nautobot.extras.models import CustomField, Role, SecretsGroup, SecretsGroupAssociation, Status
1720
from nautobot.ipam.models import Namespace
1821
from nautobot_ssot.jobs.base import DataSource
22+
from nautobot_plugin_nornir.constants import NORNIR_SETTINGS
1923

2024
from nautobot_device_onboarding.diffsync.adapters.sync_devices_adapters import (
2125
SyncDevicesNautobotAdapter,
@@ -28,6 +32,13 @@
2832
from nautobot_device_onboarding.exceptions import OnboardException
2933
from nautobot_device_onboarding.netdev_keeper import NetdevKeeper
3034
from nautobot_device_onboarding.utils.helper import onboarding_task_fqdn_to_ip
35+
from nautobot_device_onboarding.nornir_plays.empty_inventory import EmptyInventory
36+
from nautobot_device_onboarding.nornir_plays.inventory_creator import _set_inventory
37+
from nautobot_device_onboarding.nornir_plays.command_getter import netmiko_send_commands, _parse_credentials
38+
from nautobot_device_onboarding.choices import SSOT_JOB_TO_COMMAND_CHOICE
39+
from nautobot_device_onboarding.nornir_plays.processor import TroubleshootingProcessor
40+
41+
InventoryPluginRegister.register("empty-inventory", EmptyInventory)
3142

3243
PLUGIN_SETTINGS = settings.PLUGINS_CONFIG["nautobot_device_onboarding"]
3344

@@ -614,6 +625,28 @@ def run(
614625
self.sync_vlans = sync_vlans
615626
self.sync_vrfs = sync_vrfs
616627

628+
# Check for last_network_data_sync CustomField
629+
if self.debug:
630+
self.logger.debug("Checking for last_network_data_sync custom field")
631+
try:
632+
633+
cf = CustomField.objects.get(key="last_network_data_sync")
634+
except ObjectDoesNotExist:
635+
cf, _ = CustomField.objects.get_or_create(
636+
label="Last Network Data Sync",
637+
key="last_network_data_sync",
638+
type=CustomFieldTypeChoices.TYPE_DATE,
639+
required=False,
640+
)
641+
642+
cf.content_types.add(ContentType.objects.get_for_model(Device))
643+
644+
if self.debug:
645+
self.logger.debug("Custom field found or created")
646+
except Exception as err: # pylint: disable=broad-exception-caught
647+
self.logger.error(f"Failed to get or create last_network_data_sync custom field, {err}")
648+
return
649+
617650
# Filter devices based on form input
618651
device_filter = {}
619652
if self.devices:
@@ -634,7 +667,7 @@ def run(
634667
self.logger.info("No devices returned based on filter selections.")
635668
return
636669

637-
# Log selected devices
670+
# Log the devices that will be synced
638671
filtered_devices_names = list(self.filtered_devices.values_list("name", flat=True))
639672
self.logger.info(f"{len(filtered_devices_names)} devices will be synced")
640673
if len(filtered_devices_names) <= 300:
@@ -656,5 +689,72 @@ def run(
656689
super().run(dryrun, memory_profiling, *args, **kwargs)
657690

658691

659-
jobs = [OnboardingTask, SSOTSyncDevices, SSOTSyncNetworkData]
692+
class DeviceOnboardingTroubleshootingJob(Job):
693+
"""Simple Job to Execute Show Command."""
694+
695+
debug = BooleanVar()
696+
ip_addresses = StringVar()
697+
port = IntegerVar(default=22)
698+
timeout = IntegerVar(default=30)
699+
secrets_group = ObjectVar(model=SecretsGroup)
700+
platform = ObjectVar(model=Platform, required=True)
701+
ssot_job_type = ChoiceVar(choices=SSOT_JOB_TO_COMMAND_CHOICE)
702+
703+
class Meta:
704+
"""Meta object boilerplate for onboarding."""
705+
706+
name = "Runs Commands on a Device to simulate SSoT Command Getter."
707+
description = "Login to a device(s) and run commands."
708+
has_sensitive_variables = False
709+
hidden = True
710+
711+
def run(self, *args, **kwargs):
712+
"""Process onboarding task from ssot-ni job."""
713+
ip_addresses = kwargs["ip_addresses"].replace(" ", "").split(",")
714+
port = kwargs["port"]
715+
platform = kwargs["platform"]
716+
username, password, secret = _parse_credentials(kwargs["secrets_group"]) # pylint:disable=unused-variable
717+
718+
# Initiate Nornir instance with empty inventory
719+
compiled_results = {}
720+
try:
721+
with InitNornir(
722+
runner=NORNIR_SETTINGS.get("runner"),
723+
logging={"enabled": False},
724+
inventory={
725+
"plugin": "empty-inventory",
726+
},
727+
) as nornir_obj:
728+
for entered_ip in ip_addresses:
729+
single_host_inventory_constructed, _ = _set_inventory(
730+
entered_ip, platform, port, username, password
731+
)
732+
nornir_obj.inventory.hosts.update(single_host_inventory_constructed)
733+
nr_with_processors = nornir_obj.with_processors([TroubleshootingProcessor(compiled_results)])
734+
if kwargs["ssot_job_type"] == "both":
735+
nr_with_processors.run(
736+
task=netmiko_send_commands,
737+
command_getter_yaml_data=nornir_obj.inventory.defaults.data["platform_parsing_info"],
738+
command_getter_job="sync_devices",
739+
**kwargs,
740+
)
741+
nr_with_processors.run(
742+
task=netmiko_send_commands,
743+
command_getter_yaml_data=nornir_obj.inventory.defaults.data["platform_parsing_info"],
744+
command_getter_job="sync_network_data",
745+
**kwargs,
746+
)
747+
else:
748+
nr_with_processors.run(
749+
task=netmiko_send_commands,
750+
command_getter_yaml_data=nornir_obj.inventory.defaults.data["platform_parsing_info"],
751+
command_getter_job=kwargs["ssot_job_type"],
752+
**kwargs,
753+
)
754+
except Exception as err: # pylint: disable=broad-exception-caught
755+
self.logger.info("Error During Sync Devices Command Getter: %s", err)
756+
return compiled_results
757+
758+
759+
jobs = [OnboardingTask, SSOTSyncDevices, SSOTSyncNetworkData, DeviceOnboardingTroubleshootingJob]
660760
register_jobs(*jobs)

‎nautobot_device_onboarding/nornir_plays/command_getter.py

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
"""CommandGetter."""
22

3+
import json
34
from typing import Dict
45

56
from django.conf import settings
@@ -51,7 +52,17 @@ def _get_commands_to_run(yaml_parsed_info, sync_vlans, sync_vrfs):
5152
"""Using merged command mapper info and look up all commands that need to be run."""
5253
all_commands = []
5354
for key, value in yaml_parsed_info.items():
54-
if not key.startswith("_metadata"):
55+
if key == "pre_processor":
56+
for _, v in value.items():
57+
current_root_key = v.get("commands")
58+
if isinstance(current_root_key, list):
59+
# Means their is any "nested" structures. e.g multiple commands
60+
for command in v["commands"]:
61+
all_commands.append(command)
62+
else:
63+
if isinstance(current_root_key, dict):
64+
all_commands.append(current_root_key)
65+
else:
5566
# Deduplicate commands + parser key
5667
current_root_key = value.get("commands")
5768
if isinstance(current_root_key, list):
@@ -124,6 +135,14 @@ def netmiko_send_commands(task: Task, command_getter_yaml_data: Dict, command_ge
124135
except Exception: # https://github.com/networktocode/ntc-templates/issues/369
125136
task.results[result_idx].result = []
126137
task.results[result_idx].failed = False
138+
else:
139+
if command["parser"] == "none":
140+
try:
141+
jsonified = json.loads(current_result.result)
142+
task.results[result_idx].result = jsonified
143+
task.results[result_idx].failed = False
144+
except Exception:
145+
task.result.failed = False
127146
except NornirSubTaskError:
128147
# We don't want to fail the entire subtask if SubTaskError is hit, set result to empty list and failt to False
129148
# Handle this type or result latter in the ETL process.

0 commit comments

Comments
 (0)