System test for language selector is flaky

[pablobm]

The system test at test/system/select_language_test.rb fails randomly. The causes are unclear. It should be hardened so that it runs consistently.

There was an attempt to rectify this at #6589, but it didn't solve the problem.

While working on #6424, a potential clue was observed: the test failed when running on a local devcontainer, leaving the following screenshot:

This suggests that the browser gives the focus to the first element in the list (Afrikaans) instead of leaving it on the input box. If you try this on your browser (open the selector, press tab to move the focus) then you won't be able to type in the box any more, which is consistent with the test failure.

As to what could possibly be moving the focus, no idea.

[mmd-osm]

To generate some fresh ideas, I have thrown all of the relevant code in a chat prompt, and let it figure out why the test case is flaky. I'm posting it here without adding my own comments, so you can make up your own mind. I would encourage further commenting which of the options make sense (or are simply hallucination): https://gist.github.com/mmd-osm/61e02dee826af4ddcb28c29e621dca1e

[mmd-osm]

Another option I've seen elsewhere is to run some tests slower. I've only tested it on Chrome, the snap based firefox is causing some major drama (as always). When disabling the headless mode, you can even follow along the steps. Maybe give it a try as well.

Details

Then this page has lots of other recommendations on flaky unit tests...

By the way, in Chrome Headless, other tests tend to randomly fail as well with errors such as:

Failure:
ReportUserTest#test_it_promotes_issues [test/system/report_user_test.rb:46]:
`Issue.count` didn't change by 1, but by 0.
Expected: 1
  Actual: 0

Error:
AccountDeletionTest#test_the_user_is_signed_out_after_deletion:
Capybara::ElementNotFound: Unable to find link or button "Delete Account..."
    test/system/account_deletion_test.rb:28:in 'block in <class:AccountDeletionTest>'

New file test/helpers/throttle.rb:

require "selenium-webdriver"
module ::Selenium
  module WebDriver
    module Remote
      class Bridge
        attr_accessor :speed

        alias old_execute execute
        def execute(*)
          case speed
          when :slow
            sleep 0.3
          when :medium
            sleep 0.1
          end
          old_execute(*)
        end
      end
    end
  end
end

def set_speed(speed)
  page.driver.browser.send(:bridge).speed = speed
end

Updated test:

diff --git a/test/system/select_language_test.rb b/test/system/select_language_test.rb
index 62e96c52b..f31d12583 100644
--- a/test/system/select_language_test.rb
+++ b/test/system/select_language_test.rb
@@ -1,9 +1,11 @@
 # frozen_string_literal: true
 
 require "application_system_test_case"
+require "helpers/throttle"
 
 class SelectLanguageTest < ApplicationSystemTestCase
   test "can select language when logged out" do
+    set_speed(:medium)
     visit help_path
 
     within_content_heading do
@@ -26,6 +28,7 @@ class SelectLanguageTest < ApplicationSystemTestCase
   end
 
   test "can select language when logged in" do
+    set_speed(:medium)
     user = create(:user, :display_name => "LanguageTestUser")
     sign_in_as(user)
 
@@ -54,6 +57,8 @@ class SelectLanguageTest < ApplicationSystemTestCase
     click_on "LanguageTestUser"
     click_on "Mes préférences"
 
+    set_speed(0)
+
     within_content_body do
       assert_select "Langue préférée", :selected => "français"
     end

[gravitystorm]

So my usual approach to these issues is to avoid any sleep statements whatsoever. If there's a synchronisation issue, sleeps just mask the actual problem, which is waiting for completion rather than waiting a fixed amount of time. I always find that there's some combination of laptop or CI busyness, that will find a way to occasionally take slightly longer than whatever sleep delay is set! I also find that as soon as sleep in tests becomes acceptable, they multiple like rabbits.

My other usual approach is to lean heavily on the built-in waiting that comes for free with most capybara/selenium commands. So things like visit "foo"; assert_content "bar" don't normally fail, even if the page load takes 500ms, because assert_content will intelligently wait. This means that if I reviewed the current code, my first instinct would be the exact opposite of #6589 🤔