Conversation
|
Have been using it for around 48+ hours and only encountered "websocket closed None : None error" and "websocket error 104 connection closed by peer " error otherwise all features of the code are working perfectly fine ( excluding betting and drops which are still yet to test) |
Hey, thanks for your help! The The other message is probably fine. 104s can happen for a number of reasons but if it's reconnecting without issues I wouldn't worry about it. |
|
For the None : None errors I just looked into another of my Miner which is on pubserver and noticed that actually the twitch servers was having some errors as both of the miners (one on hermes and one on pubserver) were disconnected at the same time which means that it isn't actually any code issue |
|
how to download? |
You can clone my branch here. No automated builds for this one since it's not an official release. |
I have a question: If I upload this to rander.com after I've linked it to my Twitch account, why does it ask me to link it again if I've already linked it? |
Sorry, I have no idea. |
|
If I modify this part, will it work the same way? |
|
Please try to use chatgpt for code modifying questions or just open a discussion or issue in the main repo. Also do read the readme file thoroughly it explains everything. |
|
Update: There might be an issue somewhere related to betting cuz some of the best are getting logged twice that means maybe they are being processed twice (there might be a chance that the streamer actually made 2 different predictions back to back not sure though but still thought to drop my observations here) |
Yeah, that looks like 2 different predictions with the same name happening close together. Might be worth checking the stream to see if it happens again. |
|
I checked the stream and the predictions were actually made so not any issue about that. But the disconnections are a lot frequent as compared to pubsub like i disconnected 7-8 times in a 12 hours run and the most frequent was on a gap of like 35-40 mins only |
I've seen an increase in reconnects too, not to the same degree but then I only run with a single client in the pool. I think this is something they've built into the new API to better distribute server load but no way to know for sure. As long as it reconnects without issue there should be no problem. For that 35-40 min gap, were the disconnections from something like a timeout or were they from a For reference, the where the |
|
I am on normal logging level as on debug the whole console gets flooded and it really hard to observe things but still I have changed the logging to debug only for websockets module (in the Twitchpointsminer.py file) and am still waiting for a disconnect to occur. Also while going through logs rn I found these too and fyi they didn't occurred during or near any reconnection/disconnection |
You can set the file logs to
Looks like an unrelated error, a quick issue search doesn't find that anywhere so if it happens again while using the regular miner release you should open an issue about it. |
|
I don't use that too cuz my vps doesn't have root access 🤦🏻♂️😂 |
|
AI analysis of what could be happening |
I've actually already considered, at least the accurate parts, of this and I'm not sure it's necessary to respect the keepalive timeout. It feels unnecessary to close the connection early when keepalives aren't being sent in a timely manner when it could just be server congestion. Saying this, I will give you that the Twitch Web client does seem to implement something like this so maybe it would be for the best. The bit about "initiate the keep-alive" cycle is wrong, the server is responsible for sending keepalives. All the client has to do is connect, authenticate, and subscribe. The old PubSub API used a The more frequent disconnections might just be how this API is going to be. As long as the miner is able to reconnect, does it really matter if it's disconnecting more often? I might try and hide some of the log messages if the client is told by Twitch to reconnect, it's not a case the user should worry about. |
|
Logs from Debug websocket logging |
|
That's a weird one, never seen it log the frame as an error before. From what I can tell Also, can you say where |
|
As of now these are the only logs related to disconnections but I'll switch to debug logging and will get more info about the same |
|
As of now this is all I could gather hope it helps |
|
@Gamerns10s I'm sorry but without the full stack trace for the error I can't pinpoint where it's coming from. Is there any chance you could run it on a machine where you have access to the log files? I would also need |
|
In that case I can run it on termux but it'll take some time to get the logs but I'll surely try to share it asap |
|
@mpforce1 do you have any other social media or email address where I can share you the logs file |
No, sorry. The last time this happened the user uploaded them to proton storage and posted a link here. Once I'd confirmed download they deleted the files. You could try something like that. |
|
I think at one point my device lost internet connection you can ignore that disconnection 😅 |
|
@Gamerns10s Got it. I'll let you know if I find anything. |
|
|
That's actually the same error as before, the
I think I've seen somebody else complain about that one, might be worth doing a search. That message format |
|
Been pretty solid since that last update, no issues on this end. |
|
Ahh just wanted to tell you that the error update client version has become very frequent in long run sessions (24+ hours) and some more logs you should take a look at |
|
@Gamerns10s It's hard to tell what's going on because of how you've formatted the logs. To get a better idea I'd need text from the log file or an upload of the log file. However, from what I can see it looks like you might have a poor connection between you and the Twitch servers. I still have no idea what the |
|
Yeah I can totally understand that and will try to share the logs file soon |
Sorry, I don't know much about the drops system. I'd like to keep this pr more on topic, please start a separate discussion/issue if you want help with something else. |
There was a problem hiding this comment.
Pull Request Overview
This PR implements a new integration with Twitch's Hermes WebSocket API as a replacement for the legacy PubSub WebSocket API. The implementation addresses issue #722 regarding Twitch's completion of A/B testing for the new Hermes API.
Key changes:
- Added a new Hermes WebSocket client and pool implementation with authentication and subscription management
- Abstracted PubSub message handling logic into a shared
PubSubHandlerclass to avoid code duplication - Introduced a new
use_hermesconfiguration option (defaults toFalsefor backward compatibility)
Reviewed Changes
Copilot reviewed 32 out of 32 changed files in this pull request and generated 32 comments.
Show a summary per file
| File | Description |
|---|---|
| example.py | Adds the use_hermes configuration parameter to the example |
| README.md | Documents the new use_hermes configuration option |
| TwitchChannelPointsMiner/utils.py | Adds utility functions for random ID generation, timestamp formatting, and iterable combination used by Hermes implementation |
| TwitchChannelPointsMiner/constants.py | Adds HERMES_WEBSOCKET constant and CLIENT_ID_WEB for browser client identification |
| TwitchChannelPointsMiner/classes/Settings.py | Adds use_hermes to Settings slots for global configuration access |
| TwitchChannelPointsMiner/classes/PubSub.py | Extracts message handling logic into PubSubHandler class for reuse across both WebSocket implementations |
| TwitchChannelPointsMiner/classes/entities/Message.py | Adds get_topic() method to retrieve PubsubTopic from message |
| TwitchChannelPointsMiner/classes/websocket/Pool.py | Defines abstract WebSocketPool base class for both implementations |
| TwitchChannelPointsMiner/classes/websocket/pubsub/Client.py | Renames TwitchWebSocket to PubSubWebSocket and removes unused attributes |
| TwitchChannelPointsMiner/classes/websocket/pubsub/Pool.py | Refactors legacy PubSub implementation to use shared message handler |
| TwitchChannelPointsMiner/classes/websocket/hermes/Client.py | Implements HermesClient with state management, authentication, and subscription handling |
| TwitchChannelPointsMiner/classes/websocket/hermes/Pool.py | Implements HermesWebSocketPool with client lifecycle management and reconnection logic |
| TwitchChannelPointsMiner/classes/websocket/hermes/data/*.py | Adds request/response data models and JSON encoders/decoders for Hermes protocol |
| TwitchChannelPointsMiner/TwitchChannelPointsMiner.py | Updates main class to conditionally create Hermes or PubSub pool based on configuration |
| TwitchChannelPointsMiner/classes/WebSocketsPool.py | Removed legacy monolithic WebSocket pool implementation |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| self.error = error | ||
| self.error_code = error_code | ||
|
|
||
| type Data = AuthenticateResponse.DataOk | AuthenticateResponse.DataError |
There was a problem hiding this comment.
The type statement for type aliases is only available in Python 3.12+, but this project supports Python 3.6+ according to setup.py. This will cause a SyntaxError on older Python versions. Consider using a regular assignment instead: Data = Union[AuthenticateResponse.DataOk, AuthenticateResponse.DataError] with from typing import Union.
| "pubsub": self.pubsub.to_dict() | ||
| } | ||
|
|
||
| def __init__(self, subscribe: Data, _id: str | None = None, timestamp: datetime | None = None): |
There was a problem hiding this comment.
The union type syntax with | is only available in Python 3.10+, but this project supports Python 3.6+ according to setup.py. Use Optional[str] and Optional[datetime] from typing instead: def __init__(self, subscribe: Data, _id: Optional[str] = None, timestamp: Optional[datetime] = None).
| self.__lock = threading.Lock() | ||
| """Lock to prevent improper variable access in a multithreaded context.""" | ||
|
|
||
| def topic(self, subscription_id: str) -> PubsubTopic | None: |
There was a problem hiding this comment.
The union type syntax with | is only available in Python 3.10+, but this project supports Python 3.6+ according to setup.py. Use Optional[PubsubTopic] from typing instead: def topic(self, subscription_id: str) -> Optional[PubsubTopic]:.
| return None | ||
|
|
||
| def __create_new_client( | ||
| self, auth: TwitchLogin, topics: list[PubsubTopic], index: int | None = None |
There was a problem hiding this comment.
The generic list syntax list[...] is only available in Python 3.9+, but this project supports Python 3.6+ according to setup.py. Use List[PubsubTopic] from typing instead: from typing import List and topics: List[PubsubTopic].
| def __repr__(self): | ||
| return simple_repr(self) | ||
|
|
||
| def __init__(self, _id: str | None = None, timestamp: datetime | None = None): |
There was a problem hiding this comment.
The union type syntax with | is only available in Python 3.10+, but this project supports Python 3.6+ according to setup.py. Use Optional[str] and Optional[datetime] from typing instead: def __init__(self, _id: Optional[str] = None, timestamp: Optional[datetime] = None).
| return _alphabet_base_36[value] | ||
| elif value < 62: | ||
| return _alphabet_base_36[value - 26].upper() | ||
| elif value > 62: |
There was a problem hiding this comment.
Logic error in the mapping function: when value == 62, the function returns "-", but when value > 62 (which would be 63 since value is masked to 6 bits with value &= 63), it returns "_". This should be elif value == 62: instead of elif value > 62: since the only remaining case is exactly 62 after checking < 36 and < 62.
There was a problem hiding this comment.
This was just a direct translation of the js client code into python. It seems to work for me, when I run it locally it generates strings that contain _s and -s.
| else: | ||
| logger.error(f"{client.describe()} - Exception raised for response: {response}", exc_info=e) | ||
| HermesClient.on_error(client, e) | ||
| except (json.JSONDecodeError, ValueError) as e: |
There was a problem hiding this comment.
[nitpick] Inconsistent logging approach: When handling JSON decode errors (line 375), on_error is called, but when handling exceptions in message processing (line 374), on_error is also called. However, the JSON decode error already logs the error before calling on_error, while the message processing exception logs with exc_info. Consider standardizing the error logging approach.
| except (json.JSONDecodeError, ValueError) as e: | |
| except (json.JSONDecodeError, ValueError) as e: | |
| logger.error(f"{client.describe()} - JSON decode error for message: {message}", exc_info=e) |
There was a problem hiding this comment.
There's some merit to standardising the logging here. The reason it's like this is to mimic, as close as possible, the PubSub client's behaviour. However, looking at it, the "Exception raised for topiclog is in the newPubSub` handler so we can probably drop that. I'll look into that.
| # try: | ||
| # cookie_file_path = os.path.join("cookies", f"{username}.pkl") | ||
| # if os.path.exists(cookie_file_path): | ||
| # os.remove(cookie_file_path) | ||
| # logger.info(f"Deleted outdated cookie file for user: {username}") | ||
| # else: | ||
| # logger.warning(f"Cookie file not found for user: {username}") | ||
| # except Exception as e: | ||
| # logger.error(f"Error occurred while deleting cookie file: {str(e)}") |
There was a problem hiding this comment.
This comment appears to contain commented-out code.
| # try: | |
| # cookie_file_path = os.path.join("cookies", f"{username}.pkl") | |
| # if os.path.exists(cookie_file_path): | |
| # os.remove(cookie_file_path) | |
| # logger.info(f"Deleted outdated cookie file for user: {username}") | |
| # else: | |
| # logger.warning(f"Cookie file not found for user: {username}") | |
| # except Exception as e: | |
| # logger.error(f"Error occurred while deleting cookie file: {str(e)}") |
| @@ -0,0 +1,3 @@ | |||
| from .Pool import WebSocketPool | |||
| from .hermes import * | |||
There was a problem hiding this comment.
Import pollutes the enclosing namespace, as the imported module TwitchChannelPointsMiner.classes.websocket.hermes does not define 'all'.
There was a problem hiding this comment.
I don't know if this is actually a problem, I'm not familiar enough with python module exports.
There was a problem hiding this comment.
I don't think this is a problem because the .hermes import should only include things exported from the hermes/__init__.py file and that's just some specific imports from hermes/Client.py and hermes/Pool.py.
| @@ -0,0 +1,3 @@ | |||
| from .Pool import WebSocketPool | |||
| from .hermes import * | |||
| from .pubsub import * | |||
There was a problem hiding this comment.
Import pollutes the enclosing namespace, as the imported module TwitchChannelPointsMiner.classes.websocket.pubsub does not define 'all'.
| from .pubsub import * | |
| # TODO: Replace with explicit imports from pubsub, e.g.: | |
| # from .pubsub import SomeClass, some_function |
mpforce1
force-pushed
the
hermes
branch
2 times, most recently
from
7bbcab5 to
d9495f2
Compare
|
@rdavydov I've cleaned up the commit history and updated the minimum python version in the docs/dockerfile to 3.12. I feel it's ready to be merged so I've brought it out of draft. |
Implements a WebSocket client and pool for the Hermes API. Abstracts some of the handling for PubSub Messages. Adds the "use_hermes" Setting to allow the use of the Hermes WebSocket API instead of the PubSub WebSocket API. Defaults to False. Renamed WebSocket classes and moved them to new websocket module.
|
What's the state of this? I'm testing this currently and doesn't seem to be working very well. Granted this is still version 2.0.3 and not the latest 2.0.4. It's running super slow and the channels don't get processed correctly. Lots of errors. Currently I'm only able to collect RAID points since the rest doesn't work. |
|
For me it's working perfectly fine I just get a few connection reset by peer (maybe due to my server's network) sometimes and the drops aren't automatically being claimed like they are watched for the required time but aren't claimed from inventory until I restart the bot manually. Everything else is working flawlessly |
That's strange, it's been a bit since I used this branch since I've been working on other features so I can't say if there's been some change that's made it more unstable. However, when I last left it we seemed to be in a decent state and running it quickly now I don't see any issues. Version 2.0.4 shouldn't really make a difference to this because that change is focused on fixing something in the watch loop not the WebSocket. I actually rebased onto 2.0.4 locally a while ago but never pushed up the change, I'll just do that now. If you're running the miner on the same setup you've been testing my other branches (thanks a lot btw 😄) it's possible there's some sort of contamination between project files or python packages that's causing the issues you're seeing. Could you please give me some more context for your issues? When you say it's running slow do you mean load time? Can you show me an example of the errors you're seeing? |
|
My testing repos are stored in different folders and each one has its own It started spitting some errors right at the start and it took a while before actually showing something in the output. After that it only showed onlines, offlines and raids. No points were actually collected. And it stayed like that for the whole run. Errors like these: |
That sounds perfect 😄
It took a while to output something because it took a few hours to load all of the data for the 1000+ streamers you were testing with.
These errors are unrelated to changes on this branch and suggest a temporary issue with your DNS.
This sort of error I can see in your logs that all requests to As a follow up, I ran the miner with around 10 streamers all yesterday with no issues. Do you get the same issues if you run the base miner with 1000+ streamers? Do you get the same problems if you run this branch with a much smaller amount of streamers, 10s rather than 100s? |
|
So I ran it again with your latest force-push and everything is working. No errors whatsoever, besides the occasional I don't know what happened but 2.0.3 was not working at all.
It could be that he had made local changes but forgot about it, maybe that's why it was working fine for him. I cloned this branch fresh and did no changes. The only thing that I modified was the
It didn't. It only collected RAID points. |
My best guess is a temporary network issue but it's hard to say.
I think they've been doing some work on a private fork, so yeah could be.
Oh my bad, that makes perfect sense then. Raids use a different endpoint so that one must not have been having that 404 issue. |
8 participants
Description
Implements an integration with the new Hermes API. As per issue #722 it seems like Twitch has finally finished A/B testing the new Hermes API. This PR adds a new WebSocket based integration with this API. It acts as a drop in replacement for the older PubSub WebSocket integration.
As part of the implementation some of the handling of PubSub Messages has been abstracted. This was done to avoid duplicating the handling logic in both integrations.
I've added a new setting to the miner constructor
use_hermes. It defaults toFalsefor now to avoid changing current behaviour for most users. In the future this might be removed if Twitch drops the old PubSub API.I've opened this PR as draft as I'm currently seeing how it acts in the long term. This is because the reconnection logic in the new integration is different and I want to make sure it works as designed. To that end, it'd be really useful if some other community members could try out this branch and see if they run into any issues. In particular, I'd like to know:
user_hermestoFalseor omitting it from the configuration.I've tested all of this locally but more users testing in more varied setups should help catch any edge cases.
Fixes #722
Type of change
How Has This Been Tested?
Tested on my local setup. I'm currently running a long running test to check reconnection logic. However, my test setup only connects to a few streamers.
Checklist: