How to Prevent Flaky Unit Tests by Managing Test Preconditions

Unstable test cases make daily development painful: failure notifications are unhelpful, real bugs can be missed, and code changes may be unfairly blamed. Many factors cause test instability, even for unit tests.

Example of an unstable unit test:

def CreateGapLease(self):
  data_file = open('/leases/gap', 'w+')
  data_file.write(contract_data)
  data_file.close()

def testCreateGapLease(self):
  contract_writer.CreateGapLease()
  self.assertEqual(ReadFileContents('/leases/gap'),
         contract_data)

If /leases/gap already exists and contains data, the test fails. This illustrates a common problem: assuming preconditions are correct, which also applies when a database is expected to contain proper information.

When another test also uses the same file /leases/gap , the test will fail again.

Always check your assumptions. If you really need to use real resources, clear the file before the test runs:

def testCreateGapLease(self):
  if os.path.exists(lease_file):
    RemoveFile(lease_file)
  ...

However, this does not completely eliminate flakiness. If the path is an NFS mount or another test writes to it, the test may still fail unexpectedly.

Best practice: use unique resources. Refactor the function to accept a custom lease path:

def CreateGapLease(self, lease_path=None):
  if lease_path is None:
    lease_path = '/leases/gap'
  ...

The production code can call the function normally, while the unit test supplies a different file path:

def testCreateGapLease(self):
  lease_file = os.path.join(FLAGS.test_tmpdir, 'gap')
  contract_writer.CreateGapLease(lease_path=lease_file)
  self.assertEqual(ReadFileContents(lease_file), contract_data)

For faster execution, it is advisable to use a mock file system to completely avoid disk access.

Metadata

Publication Date: April 17, 2008

Original Author: Unknown

Original Link: https://testing.googleblog.com/2008/04/tott-avoiding-flakey-tests.html