Develop and implement a probabilistic flakiness score for each test as outlined in this article from Facebook engineering. It provides a reliable real time metric that can provide insight into the health of individual tests in a CI pipeline and provide engineers information on where to focus there efforts in updating tests.