CFE Declaration Randomly Reverts to whats in /config/cloud/cfe_config.json

[tigerdog6768]

Re-posting issue as previous attempt does not show in issue list.

Versions:
F5 v17.1.2
CFE v2.1.3

I have several pairs of LTMs in AWS using the CFE for both route and address failover. When the declaration is correct we have no issues however I have come across several instances where the declaration has reverted. It reverts to whats stored in /config/cloud/cfe_config.json. In the cfe_config.json it has the correct the tags however the Scoping address under the failover routes is a generic "192.0.2.0/24" with no scoping name and the defaultNextHopAddresses are also incorrect, they are the right /16 but the last two octets I dont think have been used before with these F5s.

Last occurrence of this:
5/16 - Upgrade to v17.1.2, upgrade CFE to v2.1.3, POST declaration, test failover both ways, no issues
6/24 - Check CFE declaration and it has reverted to the cfe_config.json config
In the restnoded log there have been no POST declares since 5/16 when I re-declared after upgrading the CFE. There has been no other maintenance or failovers since 5/16. Some where between 5/16 and 6/24 the CFE reverted on its own to the best I can tell.

This has happened several times on several different LTM pairs in AWS, I am not able to find a pattern. Obviously its concerning if this can happen and I dont catch it before a failover event and it causes an outage.

Has anyone else experienced this and are there any other logs I can review? What is the function of the cfe_config.json and why would it be reverting to it?

Thanks in advance!

[mikeshimkus]

Hi @tigerdog6768, /config/cloud/cfe_config.json is not created or used by CFE. Do you know how that file got there? I recommend opening a case with F5 support on this.

[tigerdog6768]

Hello @mikeshimkus

I do not but it exists on all instances that use the CFE. In speaking with our solutions engineer their recommendation was to create a Github issue. I can certainly submit a case in addition to this.

[tigerdog6768]

My case is still in progress but I have done a bunch of testing and narrowed this down. Its caused by a combination of using manual config sync and F5s deployed via AWS CFT (possibly only older versions, 6.0.0).

Essentially during an upgrade the user-data of the EC2 instance gets triggered and runs the cloud-init process. When the device group is set to manual sync there is a script "/config/cloud/aws/cfe-config.sh" that gets called that makes POST /declare to http://localhost:8100/ with what ever is in "/config/cloud/cfe_config.json" as the declaration body. In my testing this sometimes does not happen for up to 15 minutes after the instance reboots. Within that 15 minutes if I do a GET /declare, the declaration is correct. However after that 15 minutes, and after the cfe-config.sh script runs and I do a GET /declare, the declaration is incorrect and matches what is in cfe-config.json. So depending on timing, the declaration looks OK and we complete our upgrade, only to later notice that it reverted. In my testing I do not see this same behavior when auto sync is used, the process stops before the cfe-config.sh is called.

Still working with support on specifics as to why this is occurring and if its specific to the older CFT we used. Also looking into if manual sync is actually needed on the instances where this is occurring.