[Feature Request] Graceful handling & stale-data fallback for HTTP data sources

[stefanschlipfi]

What would you like to be added?

Thanks a lot for the awesome project crowdsec.

Currently, when using a data source with a source_url in a parser or whitelist, a failure to fetch the remote file (e.g., 404, DNS issues, or timeout) causes the CrowdSec Hupupdate Service process to fail.
This is problematic for automated updates (cscli hub update && cscli hub upgrade), as it can leave the engine in an inconsistent state or prevent a restart.

Proposed Solution: I propose a more resilient approach for remote data sources:

• Stale Data Fallback: If the HTTP download fails, CrowdSec should log a warning but continue to use the existing local version of the file (dest_file) if it exists.

• Non-Blocking Update: The hub update or service reload should not return an error code if the failure is limited to a remote data source fetch and a local copy is available.

• Notification Support: Trigger an alert (which can then be handled by the existing notification system, e.g., email, Slack, HTTP plugin) to inform the administrator that the data source update failed and stale data is being used.

Thanks a lot.

Why is this needed?

This is problematic for automated updates (cscli hub update && cscli hub upgrade), as it can leave the engine in an inconsistent state or prevent a restart.

[github-actions]

@stefanschlipfi: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions]

@stefanschlipfi: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind bug
• /kind packaging

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[LaurenceJJones]

1. Yes makes sense, if you have an example of the current behaviour but I guess it moves the file to a .bak but then doesnt move it back and keeps an empty file?

   tried replicating in [hub]: fallback if data_url encounters error #4260 but was not successful can you explain more what you are seeing as all tests show cscli when encountering error with revert file back to originate pre update state

2. Not really, it is an error and is useful for sys admins that have a notification already setup when a service fails

3. Simply no, we dont want to use alerts as a way to get general system alerts, these are simply for threats given an IP or a PID if you use the auditd collection for example.

We should maybe have a guide for getting notifications when the hub service fails by using the OnFailure systemd property which can execute a bash script so can generally hook into any notification system. As our notification system is heavily tied to alerts and decisions creating a new ecosystem for this would take too much resources when systemd already supports it.

Edit: talking to the team internally some of us wanted to do something like notifications for crowdsec service itself, so taking back to simply no part but we all agree this is a big task that will take sometime and resources to which ourselves do not have the time to take currently.

[stefanschlipfi]

@LaurenceJJones thanks for your quick response.
I have verified this, and indeed, the file persists with its last known good content if the update fails. ( /var/lib/crowdsec/data/[filename] )

However, I have a follow-up question regarding the execution flow:

If I have multiple data sources or multiple items to update, does a single HTTP failure (e.g., a 404 on one source) cause a panic exit for the entire cscli hub update/upgrade process?

Specifically:

• Will the remaining unrelated data sources still be updated, or is the loop interrupted?
• Does cscli return a non-zero exit code even if the other updates were successful?

[LaurenceJJones]

Will the remaining unrelated data sources still be updated, or is the loop interrupted?

Haven't dug into the code, but I am pretty sure the first err will interrupt the loop.

Does cscli return a non-zero exit code even if the other updates were successful?

It will hit the first err and return a non-zero exit code, but technically that would leave some sources in sync and some not, I need to look at the hub systemd unit but most likely that would leave crowdsec in memory out of sync with disk but then this could be risky, as for example a parser update comes down a transient issue happens and the feed is cut short. We cannot parse the yaml to a parser configuration so it then risks crowdsec being stuck in a restart loop.

However, there is some action points we can take here, however, how many custom data urls are you using?

[stefanschlipfi]

Thanks for the honest assessment. The risk of a restart loop due to out-of-sync configurations is exactly what I'm worried about.

Regarding your question: I am planning to use a handful of external feeds (e.g., specific service whitelists like Siteimprove and maybe 2-3 specialized community blocklists). It's not a massive amount, but even with just a few sources, the 'all-or-nothing' approach feels a bit risky for a stable setup.

If one external source has a transient issue, I’d prefer the system to remain functional with the 'stale' local data for that specific source while successfully updating everything else.

I'm glad to hear you're considering some action points here. A more granular error handling (continue on error + non-zero exit code at the very end) would already be a huge improvement for the overall robustness of the hub.

[LaurenceJJones]

Regarding your question: I am planning to use a handful of external feeds (e.g., specific service whitelists like Siteimprove and maybe 2-3 specialized community blocklists). It's not a massive amount, but even with just a few sources, the 'all-or-nothing' approach feels a bit risky for a stable setup.

For whitelists if they provided a range or list of IP's you should use the new allowlist feature it just overall better to handle this (if they provided useragents or whatever then yeah file based whitelists are only way to go), you can script a cronjob that can execute the cscli commands. For blocklists you can ingest them as decisions via cscli decisions import feature docs.

[stefanschlipfi]

Thanks for the pointer! Regarding the allowlist feature: it's definitely interesting, but my main challenge is that I'm dealing with dynamic data.

To clarify why I'm using the file-based approach via source_url:

• User-Agents are not enough: I can't rely solely on User-Agents as they are easily spoofed. IP-based validation is a requirement for my setup to ensure proper security.
• Multiple Dynamic Sources: I have several groups of users/contributors who maintain their own IP lists via their own HTTP feeds. They update these lists independently when their infrastructure changes.

Using the data section with source_url is the most 'elegant' way to keep these in sync without building a wrapper of external cron jobs and scripts for every single source.