exporter: track the source type node count #129
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The global "seen" and "duplicates" variables are used in a sub function. Therefore it is not necessary to declare them global in the parser functions. Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
As we already use classes just use an appropriate initializer to avoid initializing per parser. Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
Adding a "Collections summaries" debug output line as follows: ``` [...] 2026-01-20 02:56:04 [info ] Collections summaries domain_sum=31569 model_sum=37589 version_sum=34872 2026-01-20 02:56:04 [info ] Summary duplicate=82 unique=39152 ``` This allows to more easily and early on check that a change likely/hopefully has no regression. Without needing to manually sum things up from the gluon-census.prom file, without needing to wait for rendered Grafana graphs. Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
Currently we only consider nodes which have a base version string which has which a typical Gluon release version string prefix. However there are several nodes / communities which have a different format, for instance because they didn't import the Gluon tags or because they copied the repository without Git metadata. This restructuring allows to add more pattern matches later. This also adds a new label "vtype", currently defaulting to a value of "gluon-base", intended for the current behaviour of matching nodes with an expected Gluon version tag prefix. Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
This adds the following ~1132 nodes:
gluon_base_total{base="gluon-unknown",community="Erfurt",version="gluon-unknown",vtype="gluon-unknown"} 25.0
gluon_base_total{base="gluon-unknown",community="Harz",version="gluon-unknown",vtype="gluon-unknown"} 803.0
gluon_base_total{base="gluon-unknown",community="Ingolstadt",version="gluon-unknown",vtype="gluon-unknown"} 1.0
gluon_base_total{base="gluon-unknown",community="Rhein-Sieg",version="gluon-unknown",vtype="gluon-unknown"} 303.0
"(gluon-)unknown" is output and baked into a firmware by Gluon's
./scripts/getversion.sh if no .git repository is present.
Track these nodes with a 'vtype="gluon-unknown"' label.
Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
This adds the following ~114 nodes:
gluon_base_total{base="gluon-ff0c72f",community="Erfurt",version="gluon-ff0c72f",vtype="gluon-commitid"} 55.0
gluon_base_total{base="gluon-d2a9cc3",community="Erfurt",version="gluon-d2a9cc3",vtype="gluon-commitid"} 3.0
gluon_base_total{base="gluon-031a835",community="Erfurt",version="gluon-031a835",vtype="gluon-commitid"} 7.0
gluon_base_total{base="gluon-b9a621f",community="Erfurt",version="gluon-b9a621f",vtype="gluon-commitid"} 3.0
gluon_base_total{base="gluon-2ea83c4",community="Erfurt",version="gluon-2ea83c4",vtype="gluon-commitid"} 6.0
gluon_base_total{base="gluon-d0f43ae",community="Erfurt",version="gluon-d0f43ae",vtype="gluon-commitid"} 10.0
gluon_base_total{base="gluon-f1f7f61",community="Südholstein",version="gluon-f1f7f61",vtype="gluon-commitid"} 30.0
"(gluon-)$commitid" is output and baked into a firmware by Gluon's
./scripts/getversion.sh if a .git repository is present, but without any
annotated Git tags. Track these nodes with a 'vtype="gluon-commitid"' label.
Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
This adds the following ~29 nodes:
gluon_base_total{base="gluon-2019-09-27",community="Halle",version="gluon-2019-09-27",vtype="gluon-custom"} 1.0
gluon_base_total{base="gluon-2022.1+mwu1",community="Mainz/Wiesbaden",version="gluon-2022.1+mwu1",vtype="gluon-custom"} 1.0
gluon_base_total{base="gluon-15-05-2018",community="Münsterland",version="gluon-15-05-2018",vtype="gluon-custom"} 2.0
gluon_base_total{base="gluon-experimental-2025-05-27",community="Stuttgart",version="gluon-experimental-2025-05-27",vtype="gluon-custom"} 13.0
gluon_base_total{base="gluon-experimental-2025-05-18",community="Stuttgart",version="gluon-experimental-2025-05-18",vtype="gluon-custom"} 5.0
gluon_base_total{base="gluon-experimental-2025-01-06",community="Stuttgart",version="gluon-experimental-2025-01-06",vtype="gluon-custom"} 1.0
gluon_base_total{base="gluon-experimental-2025-10-02",community="Stuttgart",version="gluon-experimental-2025-10-02",vtype="gluon-custom"} 1.0
gluon_base_total{base="gluon-experimental-2025-09-19",community="Stuttgart",version="gluon-experimental-2025-09-19",vtype="gluon-custom"} 1.0
gluon_base_total{base="gluon-experimental-2025-10-15",community="Stuttgart",version="gluon-experimental-2025-10-15",vtype="gluon-custom"} 1.0
gluon_base_total{base="gluon-experimental-2025-05-08",community="Stuttgart",version="gluon-experimental-2025-05-08",vtype="gluon-custom"} 1.0
gluon_base_total{base="gluon-experimental-2025-10-23-base-28-g423258a+",community="Stuttgart",version="gluon-experimental-2025-10-23-base-28-g423258a+",vtype="gluon-custom"} 1.0
gluon_base_total{base="gluon-experimental-2024-11-24-4-g4b02790",community="Stuttgart",version="gluon-experimental-2024-11-24-4-g4b02790",vtype="gluon-custom"} 1.0
These were likely created via a custom .scmversion in the Gluon
repository / via a GLUON_VERSION environment variable.
Track these nodes with a 'vtype="gluon-custom"' label.
Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
This adds the following ~1211 nodes:
gluon_base_total{base="Debian",community="Bremen",version="Debian",vtype="foreign"} 3.0
gluon_base_total{base="NixOS",community="Darmstadt",version="NixOS",vtype="foreign"} 149.0
gluon_base_total{base="Debian",community="Chemnitz",version="Debian",vtype="foreign"} 2.0
gluon_base_total{base="Debian",community="Altdorf",version="Debian",vtype="foreign"} 11.0
gluon_base_total{base="Ubuntu",community="Aachen",version="Ubuntu",vtype="foreign"} 20.0
gluon_base_total{base="Debian",community="Aachen",version="Debian",vtype="foreign"} 20.0
gluon_base_total{base="Debian",community="Einbeck",version="Debian",vtype="foreign"} 2.0
gluon_base_total{base="Ubuntu",community="Emskirchen",version="Ubuntu",vtype="foreign"} 3.0
gluon_base_total{base="Debian",community="Essen",version="Debian",vtype="foreign"} 1.0
gluon_base_total{base="Ubuntu",community="Essen",version="Ubuntu",vtype="foreign"} 1.0
gluon_base_total{base="Debian",community="Dreiländereck",version="Debian",vtype="foreign"} 95.0
gluon_base_total{base="OpenWrt 18.06.2 r7676-cddd7b4c77",community="Dreiländereck",version="OpenWrt 18.06.2 r7676-cddd7b4c77" vtype="foreign"} 1.0
gluon_base_total{base="Ubuntu 20.04",community="Dresden",version="Ubuntu 20.04",vtype="foreign"} 3.0
gluon_base_total{base="Debian 12",community="Dresden",version="Debian 12",vtype="foreign"} 18.0
gluon_base_total{base="Debian 11",community="Dresden",version="Debian 11",vtype="foreign"} 7.0
gluon_base_total{base="OpenWrt 22.03.7",community="Dresden",version="OpenWrt 22.03.7",vtype="foreign"} 437.0
gluon_base_total{base="OpenWrt 18.06-SNAPSHOT r0+8073-5625f5bc36",community="Dresden",version="OpenWrt 18.06-SNAPSHOT r0+8073-5625f5bc36",vtype="foreign"} 223.0
gluon_base_total{base="OpenWrt 21.02.3",community="Dresden",version="OpenWrt 21.02.3",vtype="foreign"} 151.0
gluon_base_total{base="OpenWrt 21.02.1",community="Dresden",version="OpenWrt 21.02.1",vtype="foreign"} 1.0
gluon_base_total{base="OpenWrt 22.03.6",community="Dresden",version="OpenWrt 22.03.6",vtype="foreign"} 14.0
gluon_base_total{base="OpenWrt 22.03.5",community="Dresden",version="OpenWrt 22.03.5",vtype="foreign"} 31.0
gluon_base_total{base="OpenWrt 21.02.2",community="Dresden",version="OpenWrt 21.02.2",vtype="foreign"} 3.0
gluon_base_total{base="Freifunk Dresden 24.10.2",community="Dresden",version="Freifunk Dresden 24.10.2",vtype="foreign"} 12.0
gluon_base_total{base="OpenWrt 19.07.6",community="Dresden",version="OpenWrt 19.07.6",vtype="foreign"} 1.0
gluon_base_total{base="Ubuntu 22.04",community="Dresden",version="Ubuntu 22.04",vtype="foreign"} 1.0
gluon_base_total{base="Freifunk Dresden 24.10.0-rc5",community="Dresden",version="Freifunk Dresden 24.10.0-rc5",vtype="foreign"} 1.0
These are nodes which provide respondd data to meshviewer etc., but
which are not Gluon nodes. Track these nodes with a "vtype=foreign"
label. Track these nodes with a 'vtype="foreign"' label.
Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
This adds the following ~30 nodes:
gluon_base_total{base="undefined",community="Frankfurt",version="undefined",vtype="undefined"} 1.0
gluon_base_total{base="undefined",community="Mainz/Wiesbaden",version="undefined",vtype="undefined"} 1.0
gluon_base_total{base="undefined",community="München",version="undefined",vtype="undefined"} 15.0
gluon_base_total{base="undefined",community="Nordhessen",version="undefined",vtype="undefined"} 14.0
These nodes provide a node_id but don't provide a .firmware.base
or .nodeinfo.software.firmware.base version.
Track these nodes with a 'vtype="undefined"' label.
Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
It currently seems a bit wild regarding who sets or doesn't set the domain_code and/or site_code. So it seems that for now we would need to consider the tuple of (domain_code, site_code) as a unique identifier of a broadcast domain of a community, with each potentially being empty. Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
The amount of nodes with an unknown model type can be interesting to know, too. Therefore tracking those as well. Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
We don't need to catch exceptions for the whole parser anymore now. It is sufficient to narrow the KeyError exception to fetching the node_id now. Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
The total counts in the prometheus collectors should match up with our global node counting now. Check if this this the case and output an error message otherwise. Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
Add a new common parser function to avoid duplicate per parser code. Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
Prepare to not only track gluon vs. non-gluon versions but also for gluon vs. non-gluon for the domain and model counts. This way we will also potentially make the "vtype" etc. available to other collections. Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
Track non-gluon models and domains in separate collections. Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
Contributor
Author
|
Rebased onto #120. |
T-X
force-pushed
the
pr-source-type-count
branch
2 times, most recently
from
ecbc72e to
1dad3b5
Compare
Move the gauge creation and updating functions, as well as result logging functions, into separate, new functions each. This is to reduce the complexity of the main() function. Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
This allows us to track the number of nodes from a specific source type, i.e. "meshviewer", "meshviewer (old)", "nodes.json v1" and "nodes.json v2" in prometheus. This could especially be useful to know which source types we might remove from our code (e.g. "meshviewer (old)" currently has no users). Signed-off-by: Linus Lüssing <linus.luessing@c0d3.blue>
T-X
force-pushed
the
pr-source-type-count
branch
from
1dad3b5 to
259727b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This allows us to track the number of nodes from a specific source type, i.e. "meshviewer", "meshviewer (old)", "nodes.json v1" and "nodes.json v2" in prometheus.
This could especially be useful to know which source types we might remove from our code (e.g. "meshviewer (old)" currently has no users).