~80k address list

[mskarbek]

I have a firewall address list with ~80k addresses in it (use case: dropping traffic to some countries based on IP2Location database). What would be the most efficient way to migrate that portion of configuration to Terraform? Currently, I just generate a single RouterOS script and execute it each time I need to update this.

[vaerh]

Hi!
What kind of migration is required? Transferring current rules to HCL or importing existing states?

[mskarbek]

I don't need to import, I can delete the existing address list and recreate fresh from clean HCL if that is the easier route to take.
But as fair as I understand, looping over a variable list with addresses in HCL would trigger double the size of the list of rest requests to the router. First, the provider checks the state and then adds each address separately in a new request. Which means that deployment of the configuration will be relatively taxing on the router and will take some time. Subsequently, each new configuration change will trigger a state check and ~80k requests to the router even though no change to the list was made. Is there a way to add a list without looping over each address?

Take for example this HCL:

variable "drop_list" {
    type    = list(string)
    default = [
        "45.115.216.0/22",
        "45.116.20.0/22",
        "45.117.152.0/23",
        "45.121.32.0/22",
        "45.123.220.0/22",
        "45.139.188.0/22",
        "49.126.0.0/16",
        "49.236.212.0/22",
        "49.244.0.0/16",
        "57.73.208.0/23",
    ]
}

resource "routeros_ip_firewall_addr_list" "drop_list" {
    count = length(var.drop_list)
    address = var.drop_list[count.index]
    list    = "drop_list"
}

Simple plan after executing this HCL takes ~20s and that's only 10 addresses:

$ time terraform plan
routeros_ip_firewall_addr_list.drop_list[4]: Refreshing state... [id=*13CCE]
routeros_ip_firewall_addr_list.drop_list[2]: Refreshing state... [id=*13CD2]
routeros_ip_firewall_addr_list.drop_list[1]: Refreshing state... [id=*13CCD]
routeros_ip_firewall_addr_list.drop_list[8]: Refreshing state... [id=*13CD6]
routeros_ip_firewall_addr_list.drop_list[6]: Refreshing state... [id=*13CD0]
routeros_ip_firewall_addr_list.drop_list[5]: Refreshing state... [id=*13CD4]
routeros_ip_firewall_addr_list.drop_list[0]: Refreshing state... [id=*13CD1]
routeros_ip_firewall_addr_list.drop_list[7]: Refreshing state... [id=*13CCF]
routeros_ip_firewall_addr_list.drop_list[9]: Refreshing state... [id=*13CD5]
routeros_ip_firewall_addr_list.drop_list[3]: Refreshing state... [id=*13CD3]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

real    0m21.094s
user    0m0.504s
sys     0m0.206s

[vaerh]

Yes, you're absolutely right. There will be a lot of queries. Not only that, the state of the resource will be queried for after each record. You will also have a large .state file.

Given that you don't need tracking for every address maybe it makes sense to combine MT and TF automation? One way is to generate a list of addresses locally, if it is loaded dynamically, and pass it to MT. Then run a script that will generate /ip firewall address-list.

https://forum.mikrotik.com/viewtopic.php?t=166293

Through TF you can create a script and schedule, upload a file.
It is probably better to clear the address list from the MT itself.

[mskarbek]

It's a shame that we can't trigger /system script run through rest and need to work around that with schedule.

[vaerh]

@mskarbek
Even though it goes against the concept of TF, but why not? I give you a gun that can be used to shoot anything without TF control.
Please write from the results, or even it would be better to put a part of the resulting scenario (without sensitive data) in the provider's guide.

resource "routeros_system_script" "script" {
  ...
  launch_trigger = timestamp()
}