Using existing VPC for NICs

[aymanelbacha-ismena]

having 8 NICs, We're trying to add all 8 internal NICs (none are external) using existing VPCs, one of them is dedicated for HA-SYNC and the remaings have ILB for each NIC
Appreciate your support to give another example on using an existing VPC as you did for VPC_protected.

[dux-fortinet]

Hi @aymanelbacha-ismena ,

Thanks for raising this issue.
I need a bit more clarity on the requirement to "add all 8 internal NICs using existing VPCs"?
Are you suggesting that the Terraform project should not create any new VPCs and ILBs, and instead utilize the existing infrastructure?

Here are two conceptual diagrams to illustrate potential implementations:

The first one is similar to the autoscale_fgt_lb_sandwich, and the Terraform project will create 8 VPCs and 7 ILBs for you.

As for the second one, the Terraform project will not create any VPCs; rather, it will configure NICs to connect with your pre-existing VPCs. (The structure of existing VPCs depends on your implementation. The structure of VPC1 and VPC2 are different in this diagram.)

I think the second diagram better fits your current needs. Do you need Internet access? Please provide further details to help us tailor an example that best fits your needs.

Thanks,
Dux

[aymanelbacha-ismena]

Hi Dux,

I realllyy appreaciate the effort of sharing the above !! really intrested the way you put this code for auto-scale
Indeed, I am trying to add/merge the code to GCP fast fabric(VPC, firewall rules as well peering already created), where it should connect to pre-existing VPCs, where their would be an option in the tfvars under the list of network interfaces for example to specify subnets and VPCs already created
So I suggest if the following can be added:

• removal of public IP and external load balancer, since it will be an internal/DC Firewall
• ILB for each NIC except for HA sync as not routes will be involved
• Custom routes where the forwarders created are the next hopes

Let me know if that answers your doubts
I'd like to share that better to make a condition on the password, since it will fail to reset since it won't conform with password policy and I have increased the timeout to 60 seconds in the function

[dux-fortinet]

Hi @aymanelbacha-ismena ,

Thanks for the prompt reply and suggestions! I will support your requests in the upcoming release.

Please correct me if I'm wrong: In your scenario, you already have 7+1 existing VPCs, and you'd like this Terraform project to create a FortiGate auto-scale group (A group of FortiGates with 8 interfaces (NICs). The number of FortiGates varies depending on the workload). The NICs of FortiGates connect to your pre-existing 8 VPCs. 7 Internal Load Balancers (ILBs) should be created by this Terraform project (Since you can't create an ILB without specifying its backend, ILBs can't be pre-existed before this Terraform project). In your 7 pre-existing VPCs (except the HA-SYNC VPC), each VPC will have a new ILB and a new route that directs all outbound VPC traffic to the ILB's frontend.

The architecture of this project should be like the second diagram in my previous comment.

I will start to work on your requests. This Github project is expected to be updated every two months, time varies depending on the workload and requests.

I will leave new comments if I encounter any issues or need further clarification.

Thanks,
Dux

[aymanelbacha-ismena]

Hi Dux,

I really appreciate your effort and prompt support since day 1.
indeed, your description mentioned above clearly describe of what we except to have in our Landing Zone.
Looking to hear from you on the status of this release since customer is looking to have it working as expected the soonest

Thanks,
Ayman

[aymanelbacha-ismena]

Hi Dux,

Having said that "This Github project is expected to be updated every two months"
We're wondering about the status of our shared request sent 2 months ago.
is there any info you'd like us to share

Thanks
Ayman

[dux-fortinet]

Hi @aymanelbacha-ismena,

Sorry for not updating recently. A new example "autoscale_fgt_as_hub" has been developed and is now in the QA process. This example is expected to be released next week (before 11/15).

Here is the preview document

Any suggestions are welcome. I will bring more code information next workday.

Thanks,
Dux

[dux-fortinet]

Hi @aymanelbacha-ismena,

The new example has been released. You can use this example to deploy a FortiGate Autoscale Group and connect your existing VPCs.

Please feel free to let me know if you have any questions regarding this example.

Best regards,
Dux

[aymanelbacha-ismena]

Hi Dux,

Thanks for sharing the updated repo.
it does support 8 NICs, yet, whenever I do backup/restore from an existing config for the policies and addresses, without using the config_script/file mentioned in the tfvar, whenever I have a new scaled device(secondary) I can see that the config is not in sync with the primary

moreover, do we need to create static route to the connector subnet

But you need to specify the corresponding route of FGTs in "config_script" or "config_file" so FGTs can reply to the Cloud Function requests from "cloud_function.function_ip_range".

Thanks for the support,
Ayman

[dux-fortinet]

Hi Ayman,

Thanks for reaching out. I'm pleased to answer your question.

[For the first configuration problem, I may need more detailed information.]
"config_script" and "config_file" are only used for initialization. Once your autoscale group is created, you can modify the primary FGT for further configuration (and please do backup/restore on this primary FGT). And the primary FGT will automatically send the configuration to the secondary FGTs. And for the newly created secondary FGT, its initial config might be defined by the "config_script" and "config_file", but then, the "Cloud Function" will tell it to sync with the primary, it may take some time (usually few minutes) for the secondary FGTs get configuration data from the primary FGT and finally have the same configuration as the primary FGT.
Both "autoscale_fgt_lb_sandwich" and "autoscale_fgt_as_hub" examples have the variable "autoscaler->cooldown_period", which allows you to specify how long (seconds) it takes for the secondary FGTs to initialize and sync configuration until it is ready to serve.

[For the "create static route to the connector subnet"]
If you mean the route to your existing VPC: the cloud function automatically creates them.
(The cloud function is a key component of both "autoscale_fgt_lb_sandwich" and "autoscale_fgt_as_hub" examples. The cloud function helps you manage primary-secondary FGTs selection, sends necessary configurations to FGTs, and licenses FGTs for you.)

If you mean the route to the cloud function:
The cloud function needs to have its only IP range and it needs the resource "google_vpc_access_connector" to connect to one of your existing VPCs.
By default, the cloud function connects to your first VPC (cloud_func_interface = "port1").
In this example document, I set (function_ip_range = "192.168.8.0/28"), so the IP range of the cloud function is "192.168.8.0/28".
Your FGTs need to route function_ip_range ("192.168.8.0/28") to cloud_func_interface("port1").
In the example document, "port1" is the default route port, and all unmatched traffic will be sent to the default route (It may not be true in your scenario), therefore, I ignored the static route for the cloud function.
The ignored route for the cloud function should be:

config router static
  edit 0
    set device port1   # port1 is the value of "cloud_func_interface"
    set gateway 10.0.0.1   # 10.0.0.1 is the gateway of your existing VPC connecting to port1
    set dst 192.168.8.0/28  # 192.168.8.0/28 is "function_ip_range"
  next
end

If your "cloud_func_interface" is port1, usually, you can ignore the above script. However, if your "cloud_func_interface" is "port2", "port3"..., please add the above script so "FGTs can reply to the Cloud Function requests from "cloud_function.function_ip_range"".

Once you deploy the Terraform project, if your FGT password is still its "Instance Id" and not the value you specified in "fgt_password", it may indicate the cloud function can't connect with the FGTs, and it is possible because the above route static is missing.

Regards,
Dux

[aymanelbacha-ismena]

hi Dux,

Thanks for your reply.
even though there's no static route, instance's pass is updated
I added the route to the VPC connector trying to solve the syncing issue whenever new secondary device is added, I dont see the route added neither on the primary nor the secondary, thus, still config is not synced
Do I still have something missing

The below is used for the config
config_script = <<EOF
config router static
edit 100
set device port1 # port1 is the value of "cloud_func_interface"
set gateway 10.80.4.33 # 10.0.0.1 is the gateway of your existing VPC connecting to port1
set dst 192.168.8.0/28 # 192.168.8.0/28 is "function_ip_range"
next
end
EOF

Moreover, I am using existing ILB so that I don't recreate routes, while changing the backend to the new instance group,
I am only updating the env variables list of ILB and redeploy.

A screenshot is attached showing the logs for both primary and secondary

[dux-fortinet]

Hi @aymanelbacha-ismena,

[About Cloud Function]
Based on the log screenshot, the Cloud Function works well, it can connect your FGTs and send the initial configuration to your FGTs. And since you said "even though there's no static route, instance's pass is updated", the "config router static" script is optional.

[About router static]
You mentioned "I dont see the route added neither on the primary nor the secondary", do you mean you added

config_script = <<EOF
config router static
edit 100
set device port1 # port1 is the value of "cloud_func_interface"
set gateway 10.80.4.33 # 10.0.0.1 is the gateway of your existing VPC connecting to port1
set dst 192.168.8.0/28 # 192.168.8.0/28 is "function_ip_range"
next
end
EOF

But when you check the router static of the FGT, the FGT doesn't have this rule?

There are two possible reasons I could imagine:

1. The grammar of some scripts in your "config_script"/"config_file" is incorrect, and the FGT could not parse parts of those incorrect scripts, so those incorrect scripts are ignored. For example:

config system interface
...
# missing end
config router static #<- doesn't work, because you are still in the "config system interface"
...

2. Scripts in "config_script" and "config_file" can't be changed after you run terraform apply to deploy the Terraform projects. Changing the contents in "config_script"/"config_file" won't change the config of deployed FGTs. So if you try to change the "config_script" after the project has been deployed, it doesn't work.

[About sync]
Please check the port used for sync.

ha_sync_interface = "port4"

In this example, port4 is used to synchronize information.
It may be a problem if, for example, your FGTs only have 2 ports yet you set port4 as ha_sync_interface.

Please also check whether the FGTs are synchronized by checking the GUI:

The primary FGT should look like:

The secondary FGTs should look like:

And if you change the Firewall rules/route static... in the primary FGT through GUI, the secondary FGT will be changed automatically and quickly (I would say, usually, less than 5 seconds).

[Others]
You mentioned "Moreover, I am using existing ILB so that I don't recreate routes, while changing the backend to the new instance group, I am only updating the env variables list of ILB and redeploy."

This may cause a problem since the environment variable ILB is used in the FGT initialization: When one new FGT is created, the Cloud Function will read all environment variables, generate a corresponding config script, and send it to the FortiGate.
So the following steps may cause a problem:

1. New FGT is created, and the cloud function sends the config script to the FGT with the old ILB value.
2. You change the environment variable ILB to a new value and change the backend of your ILB to the new instance group.
3. This results in the config of FGT using an old ILB value, which is different from the ILB you are using in real.

I will do further research to figure out whether it is possible to use existing ILB.

Best,
Dux

[aymanelbacha-ismena]

hi Dux,

Unfortunatley after a lot of retries, the result is the same, no sync between secondary and primary.
Although route is added into both to connect to the VPC connector.
The only change I added from your tfvars, is having 8 NIC, commenting the routes, attached
as well I have updated the main.py, so the HA-sync and default route NIC are updated.
I have hardcoded the ILB_IP_LIST in function.tf as jsonencode(["10.80.4.36","10.80.4.164","10.80.4.100","10.80.4.132","10.80.4.68","","","10.90.0.68"])
the 2 unused are for mgmt and ha-sync, since they don't have ILB.
Moreover I have that the psk encyrpted pass is different on both for the HA-sync (could it be related for not syncing?)

Appreciate your support and your guidance, if you'd like we can connect on quick gmeet
terraform.txt

[dux-fortinet]

Hi @aymanelbacha-ismena,

Sorry for the late response. I was on vacation with limited work access. I'm back now.

Based on the information you provided, the Cloud Function works well.
And for the worries about "psk encrypted pass is different on both", maybe the psk pass is the same on both sides: Every time you use "show system auto-scale", the FGT will give you different encrypted data.

We need to analyze further to figure out why the sync is not working. And yes, it is a good idea to schedule a quick gmeet.
My email is dux@fortinet.com, I'm available 10 AM-6 PM PST for all business days.
Please send me emails for more private discussions and the meeting.

Best,
Dux

[aymanelbacha-ismena]

hi Dux,

Hope you had a good vacation, I assumed so especially at these times
indeed, I agree with you CF is working plus key is different due to each device having its own encryption key
I will send an invitation at 10 AM ( 8 PM GMT+2 local time)
Looking forward to our call

Thanks.
Ayman

[dux-fortinet]

The example code to deploy 8 NICs FGT autoscale group in the region "me-central2", and port6 used for sync, port7 used for mgmt:

me_cluster.tar.gz