Freetext search causes excessive requests when getting a large number of results

[dianabarsan]

Describe the bug
Simple freetext search seems to get all results for the search for every search "page" instead of getting one page at a time.

To Reproduce
Steps to reproduce the behavior:

1. Have a large instance. your search term should yield more than 100 results.
2. Make a report freetext search - don't use other filters.
3. It takes a long time to load the results.
4. Scrolling to the next results page takes equal amount of time.
5. Inspecting the console, you see many ajax requests to the server.

Expected behavior
A single request should be made for one page of results.

Video

many.nouveau.reqeusts.mp4

Environment

• App: webapp
• Version: 5.0.0

[jkuester]

Okay, I think I found the root problem. TLDR I think we accidentally left a 100 result limit in the cht-datasource "paged generator" code when it should have been updated to use the default 10000 result limit when paging uuids.... This means that when you run a freetext search and there are a lot of results, we will round-trip to the DB for ever 100 uuids that are found by the search. 😬 This does affect both online and offline users (though presumably offline users are less impacted because they have less total search results and no network latency for round-trips to Pouch).

We set the 10000 result limit as the default when getting a "page" of uuids from cht-datasource. But, shared-libs/search needs to get all uuids matching the freetext search. So, its logic just goes straight into the generator-based function that does "automatic" paging. And then spins through all results of the generator (behind the scenes the generator is fetching uuids in batches of 100...). We had to have some kind of paging here because Nouveau has a default limit of 25 if you do not set some other limit. So, the idea was to just grab 10000 at a time... 😞

The good news is that I guess this should be easy to fix since we just need to update the getPagedGenerator logic to not explicitly set the limit value. I think if we leave off that parameter, then we will just use the default limit value defined for the fetchFunction and this should give us the desired flexing between the DEFAULT_DOCS_PAGE_LIMIT of 100 vs the DEFAULT_IDS_PAGE_LIMIT of 10000....

[jkuester]

Simple freetext search seems to get all results for the search for every search "page" instead of getting one page at a time.

One thing to note (because you called this out in the issue description) is that AFICT the freetext search has never used db paging. Instead, shared-libs/search always used its "manual" paging for freetext search. Even on the pre-nouveau code (e.g. 4.22.0) the freetext requests never have ordered set and so always drop down into the manual pagination process which queries the views without any limit. The difference that was introduced in 5.0.0 was that if your search yields 1000 results, before 5.0.0 we would run one view query with no limit and get back all 1000 results and then manually return the first 50 of them. Now, after 5.0.0 we will end up running 10 round-trip Nouveau queries (for 100 results each). Then we combine all the results and manually return the first 50.

[dianabarsan]

5.0.0 we will end up running 10 round-trip Nouveau queries (for 100 results each). Then we combine all the results and manually return the first 50.

I don't think this is happening. In the video I shared, there are over 100 requests for one set of results. so it's 10.000 results.

[jkuester]

🤔 Looking more closely at the video, it appears that we may have multiple requests going through with the same cursor value, which would def be a problem. (Edit: never mind. I did some tests and saw that adjacent cursor values from Nouveau can start/end with the same chars (but have different values in the middle), so my observation of possible duplicate cursors in the video is almost certainly wrong.)

Initially, I expected the issue to be somewhere in the fetchAndFilter logic (since that code is pretty complex). When I saw the low limit value, I thought it might be a more simple problem. Now I am back to wondering if it is in the fetchAndFilter....

[jkuester]

Hmm, never mind. Digging more into the code here, it looks like for the Nouveau search we never hit the fetchAndFilter logic at all. 🤔 This concerns me a bit from a logic consistency perspective, but I think it ultimately makes our debugging here more simple because the current Nouveau flow is more straightforward....

From the client side, I have not (yet) found anything wrong with the logic that would trigger extra requests (besides the low limit). I did a local test with 15300 contacts named "Andrew Garfield" and then did a search for "Andrew". With the master code, I saw 154 requests against api/v1/contact/uuid which maps to 1 request per ~100 uuids. Then, when I removed the limit (#10622) so that it could just run with the default 10000, I saw the number of requests drop to just 2.

Now, after poking more around in the Nouveau code, I am a bit confused about one piece of the batching that is running on the server side. As far as I can tell this is not really affecting anything, but I am still not sure I understand how this should work.... 🤷 (I feel like that check is backwards and should maybe be re-running the recursionInner when we don't have DEFAULT_IDS_PAGE_LIMIT. But as things are, I don't think we would ever hit that recursion with the paged generator since we are using limit = 100 and will never get 10000 records back on the request...).