Skip to content

Various X bug fixes#435

Merged
micahflee merged 10 commits intomainfrom
432-error-indexing
Mar 4, 2025
Merged

Various X bug fixes#435
micahflee merged 10 commits intomainfrom
432-error-indexing

Conversation

@micahflee
Copy link
Member

@micahflee micahflee commented Feb 28, 2025

Fixes #432.

Also fixes #416, which I just realized was introduced in #384.

I also merged #434 and #430 into this branch, so we should review/merge those first.

For some reason, X returned a tweet (technically a RT) with entities that had urls that looked like this:

"urls": [
    {
        "display_url": "t.co",
        "url": "https://t.co/",
        "indices": [
            130,
            143
        ]
    }
],

While indexing, this now only saves URLs from tweets if all of the correct fields are present.

During indexTweets, X sometimes responds with 429 even when it is not really rated limited. This leads the indexing to stop early because it cannot find the retry button. This PR checks to make sure this is not the case before looking for the retry button, so the indexing doesn't stop early. I think this should make indexing more robust.

There were also various problems I noticed with saving DMs. When trying to save conversations it sometimes would timeout too soon and move to saving messages, so I increased the timeout. I fixed that by increasing the timeout, and also fiddling around with scrolling after the page loads to encourage it to show conversations. It seems to work.

I also added a "Enabling Clicking in Browser" link to the job status component. When you click it, you can click around in the browser during automation. If you click again, you can disable it.

Screenshot 2025-02-28 at 4 47 39 PM

It was difficult for me to reproduce, but I did get to a point when indexing DMs where there was a "retry" button that Cyd wasn't clicking and it was stuck, so this allows the user to click it themselves if they want.

@micahflee micahflee marked this pull request as ready for review March 1, 2025 00:53
Copy link
Contributor

@redshiftzero redshiftzero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@micahflee micahflee merged commit 72061ed into main Mar 4, 2025
1 check passed
@micahflee micahflee deleted the 432-error-indexing branch March 4, 2025 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

X: error indexing tweets X: error saving DM users

2 participants