gradle.com Open in urlscan Pro
2606:4700::6810:a66f  Public Scan

Form analysis
0 forms found in the DOM

Text Content

Join us for the next DPE Showdown – June 13, 2024 – 10AM PDT Save your spot

 * Solutions
   
   * * * Key Capabilities
       
       * Overview
       * Performance Acceleration & Continuity Solution Overview
       *     Build Cache
       *     Predictive Test Selection
       *     Test Distribution
       *     Performance Continuity
       * Build Scan®
       * Failure Analytics
       * Management Reporting & Insights
       * CI Build Cache & Resource Profiling
       
       * Platform Services
       
       * Performance & Scalability
       * Security
       * Integrations
     
     * * Solutions for…
       
       * DPE Engineers & Managers
       * Senior Engineers
       * Build Engineers
       * VPs & Directors of Engineering
       * CI & Platform Engineering
       * Application Engineers
       
       * Build Systems
       
       * Android
       * Bazel
       * Gradle Build Tool
       * Maven
       * Scala & sbt
     
     
     * * DPE UNIVERSITY
         
         Start your journey to Developer Productivity Engineering Mastery
         
         Get Started
 * Pricing
 * Customers
   * Commercial
   * Success Stories
   * OSS Projects
   * Profiles & Results
 * Learning Center
   * * * 
       * All Resources
       * - By Learning Objective
       * - By Journey Phase
       * - By Key Topics
     * * 
       * DPE University
       * Tour Develocity
       * Events & Training
       * ROI Calculator
       * Documentation
       * YouTube Channel
       * Blog
     
     
     * * DPE UNIVERSITY
         
         Start your journey to Developer Productivity Engineering Mastery
         
         Get Started
 * Company
   * Who We Are
   * Our Story
   * Working at Gradle
   * Our Team
   * Press & Media Center
   * Brand
 * Build Tool
 * Take the Tour

Developer Productivity Engineering Blog
Share this...

Facebook
Pinterest
Twitter
Linkedin


TOPICS


DPE Industry Explainer/Tutorial Product Announcement Customer Success


DEVELOPER PRODUCTIVITY ENGINEERING HANDBOOK

How do the world’s best software development teams create a productive and
joyful developer experience?

Download the Book


3 KEY ELEMENTS TO INCORPORATE INTO YOUR FLAKY TEST REMEDIATION APPROACH




By Trisha Gee
August 11, 2023

Flaky tests pose substantial challenges due to their unpredictable and
inconsistent nature. Effectively addressing them requires a multi-faceted
approach that involves the effective integration of strategy, process and
resource alignment, and a deep understanding of flaky test causality. This post
will walk you through this approach.

Note! This post is part of a three-part series. If you’re not sure it’s worth
remediating flaky tests, read Part 1: Seven Reasons You Should Not Ignore Flaky
Tests. Read Part 2 to understand the keys to identifying and tracking flaky
tests, called 5 Ways to Use Develocity to Identify and Manage Flaky Tests. Now,
let’s explore my multi-faceted approach to fixing flaky tests.


1. DEPLOY BEST PRACTICE STRATEGIES

Once you have identified which of your tests are flaky, you can use one of these
strategies to mitigate the problems they cause.



 1. Quarantine Flaky Tests: Isolate flaky tests to prevent them from disrupting
    the development process and distracting developers from genuine failures.
    Once quarantined, these tests can be analyzed separately, freeing developers
    to focus on legitimate failures.
 2. Improve Error Reporting: You may need more information in order to find the
    causes of the test failures. Enhancing your error reporting can
    significantly aid in handling flaky tests. This can be achieved by adding
    assertions, checking preconditions, and logging more details about the test
    environment and state.
 3. Retry with Care: While retrying can be a useful tool in identifying flaky
    tests, it’s not a strategy for solving the problem. Retrying until a test
    passes masks the intermittency and wastes resources in CI and locally.
 4. Commit to Fixing Flaky Tests: Once you’ve tracked down the flaky tests, and
    perhaps improved the error reporting and quarantined them from interfering
    with your team’s productivity, the goal should be to fix the test.

You can also read about how the Gradle Build Tool team handles flaky tests.


2. ALIGN YOUR PROCESS & RESOURCES

If you don’t want to rely on a few individuals with the discipline and
determination to fix your flaky tests, you’ll need to implement some process
changes to make sure time is allocated to fixing the problems.

When a developer commits a change that breaks a test, the developer or
developers who worked on that change usually start working on fixing that test.
This is a well-accepted approach to fixing breaking changes, but it equally
applies to tests that start to fail intermittently. 

If your application already has a number of flaky tests that aren’t owned by a
developer, you may want to schedule regular Flaky Test Days. These dedicated
sessions not only aim to decrease the number of flaky tests in your test suite,
they also emphasize the importance of addressing test flakiness, and foster a
culture of collective responsibility toward improving test reliability.


3. UNDERSTAND THE CAUSES OF FLAKY TESTS

The causes of test intermittency are varied and nuanced, as discussed by Dave
Farley in his video, 5 Reasons Automated Tests Fail, and collated in a research
paper on the impact and causes of intermittent tests. Each test may be a unique
case, but you may also find that one cause of intermittency affects multiple
tests. 

Here are some common causes of test intermittency. Note that these categories
can overlap, but considering each failure from one of these angles may lead to
identifying a fix for the failure.

1.  Concurrency, Asynchronous Programming, and Waiting: Asynchronous and
concurrent programming pose specific challenges to testing. Tests often have to
wait for events to happen before taking the next steps or may run into race
conditions in either the test code or production code. There may be
environmental factors in these failures too, since tests may time out more
frequently if the test environment is under a high load.


2. Environment, Network, and Resources: Variations in testing environments or
network conditions, as well as insufficient compute resources, can result in
inconsistent test behavior. Develocity can help you identify some of these
issues—it will show details about the environment the tests ran in so you can
compare the test results from different environments.



3. Integration Points: Tests depending on external systems or services
(integration points) may be flaky due to the unpredictable nature of these
dependencies. This includes other services from inside your organisation, as
well as third-party libraries and APIs or systems that are external to your
organisation. Integration tests against external systems are valuable since they
can tell you if your assumptions about the system are correct. However, tests
that are designed to run against systems that can change without warning should
be kept separate from the main test suite due to their inherently uncertain
behaviour. And the main test suite should protect itself from these integration
points by mocking and stubbing the expected behaviour.


4. Setup/Teardown and Test Data: Test results can only be predictable if the
start state of the test and end state of the test are also predictable. If the
tests rely on shared state, shared data, or shared resources (like a database),
this can be a contributor to intermittency in the tests. It’s key to make sure
the tests run in isolation so they don’t impact the data from other tests. Even
when the data is isolated from other tests, you may still run into unpredictable
results if your test or production code is something that’s randomly generated,
or related to date and time. You may want to inject a custom provider of random
values or date/time into your production code so that you can control these
values from the test.



5. System Behavior: While it’s easy to assume it’s some problem with the
environment or test data that’s causing an intermittent failure, sometimes the
problems lie in the production code of your application. For example, test
environments can sometimes trigger genuine race conditions in concurrent code or
uncover bugs in third-party libraries. These can sometimes be the most difficult
issues to identify but are arguably the most important reason to address flaky
tests.


CONCLUSION & NEXT STEPS

Efficient management of flaky tests is a combination of strategic actions,
process changes, and a deep understanding of root causes, all facilitated by
tools like Develocity. By weaving these elements together, your team can
effectively navigate the challenges posed by flaky tests, ensuring the delivery
of high-quality, reliable software.

Here are two videos I recommend to learn more. The first is about Develocity’s
Failure Analytics, which provides Test Failures Analytics and flaky test
management capabilities in particular. The second is about Build Failure
Analytics, which addresses non-test-related failures like compilation problems.

 * Improve Gradle & Maven Flaky Test Management with Test Failure Analytics
 * Improve Gradle & Maven Build Reliability with Build Failure Analytics


COMMUNITY

 * Twitter
 * Blog
 * LinkedIn
 * YouTube
 * Newsletters
 * Gradle.org


GET STARTED NOW

 * Solutions Overview
 * Develocity Demo (Video)
 * Start a Free Trial
 * Contact


ABOUT

 * Who We Are
 * Our Story
 * Working at Gradle
 * Our Team
 * Press & Media Center
 * Brand


© Gradle Inc. 2024

Privacy Policy | Terms | Status

We'd like to collect non-essential cookies for analytics and marketing which
involves cookies managed by third parties. You can read more about how we use
cookies in our Privacy Policy. By clicking “Accept,” below you agree to our
website's cookie use as described in our Privacy Policy and our collection of
non-essential cookies.

Reject Accept