It Doesn't Have to Be Arbitrary

by @mjgpy3 on April 07, 2022

At first glance, writing QuickCheck Arbitrary instances seems cut-and-dried. It’s simple, right? You just have the instance yield all inhabitants of a given type!

Well, what about impossible invariants we can’t enforce with the type system? How about marginal states that we know are possible but are unlikely, or highly unusual? Should such cases be possible in our Arbitrary instances and thus propagate through all our property-based tests?

At Freckle, we have a large Haskell codebase that’s rigorously tested using randomized data and property-based testing. After years of defining Arbitrary instances we’ve come up with the following guidelines to keep test overhead low and maximize the benefits of property-based tests.

A Good Arbitrary Instance

The following guidelines capture Freckle’s philosophy when writing and maintaining Arbitrary instances.

Prefer Randomized Values

This is a given; property-based testing embraces randomness. As our default strategy we want to exhaust unexpected cases as much as possible.

This post isn’t meant to be a deep dive into the benefits of testing with randomized data (for that I’d recommend Scott Wlaschin’s excellent talk). That being said, I recall a few crucial issues that I’ve caught with property-based tests in actual production code:

Avoid Random Values Where Overwhelmingly Practical

Though we prefer randomness, there are certain values we’d never like to be random by default.

For example, many of our database entities are soft-deleted (e.g. by setting a flag on the entity). Simplified a bit, a record that represents a Student may look like the following:

data Student =
  Student
  { firstName :: NameComponent
  , lastName  :: NameComponent
  , grade     :: Grade
  , createdAt :: UTCTime
  , deletedAt :: Maybe UTCTime
  }

where soft deletion is represented by a deletedAt timestamp.

softDelete :: UTCTime -> Student -> Student
softDelete now student = student { deletedAt = Just now }

A random deletedAt is impractical.

With a random deletedAt, tons of tests would have to overwrite this property to pass.

You might argue that this is a good case for the newtype override strategy and you’re right! If you’re not familiar with that strategy, that’s fine – we’ll discuss it a little later.

Don’t Include Impossible States

In the ideal world, our types would perfectly constrain what’s possible at the value-level. In practice, it’s sometimes not practical or even possible to achieve this level of fidelity. Within a record or database table, values are often implicitly interdependent. We may expect the absence of a value (e.g. Nothing) because of the presence of another value. Data is tricky.

Imagine our Student entity’s state gets more complicated. Let’s say we need to track a student’s data source and our system ensures that…

data Student =
  Student {
  -- ...
  , deletedAt :: Maybe UTCTime
  , activeDataSource:: Maybe DataSource
  }

So, since we’re keeping deletedAt as Nothing for practical concerns, our Arbitrary instance itself should now uphold the following property.

activeHasDataSource :: Student -> Bool
activeHasDataSource student =
  isNothing (deletedAt student) && isJust (activeDataSource student)

Keep states that are impossible out of Arbitrary instance for the same reasons that you default certain data: it’s practical and desirable in most tests.

Here’s our final Arbirary Student instance.

instance Arbitrary Student where
  arbitrary = do
    firstName  <- arbitrary
    lastName   <- arbitrary
    grade      <- arbitrary
    createdAt  <- arbitrary
    dataSource <- arbitrary

    pure Student
      { firstName        = firstName
      , lastName         = lastName
      , grade            = grade
      , createdAt        = createdAt
      , deletedAt        = Nothing
      , activeDataSource = Just dataSource
      }

newtype as an Override Strategy

The Arbitrary instance, though core to QuickCheck, is only the default. That’s where newtype comes in as an override strategy.

We previously discussed using a newtype for generating a deleted Student. This is an example of the newtype strategy.

newtype DeletedStudent = DeletedStudent Student

instance Arbitrary DeletedStudent where
  arbitrary = do
    student   <- arbitrary
    deletedAt <- arbitrary

    pure $ DeletedStudent student
      { deletedAt = Just deletedAt
      , activeDataSource = Nothing
      }

This gives us a coherent deleted student, which still has maximally random data, without requiring a ton of context. Nice!

The newtype pattern generally provides modeling benefits. For more information I’d recommend reading the Haskell Mini Patterns entry since the values, examples, and costs therein also apply to test-specific newtypes. One additional benefit that’s specific to test-bound newtypes is that they’re signals. They beg the question, could the production code benefit from this newtype?.

Functions as an Override Strategy

Another common strategy for overriding arbitrary values that we use throughout our codebase is to create a function that acts as a generator.

This is a nice approach because it allows you to inject arbitrary context into the generator (e.g. dependencies or monadic context that’s not Arbitrary).

The function-generator that gives us a deleted student is:

genDeletedStudent :: Gen Student
genDeletedStudent = do
  student   <- arbitrary
  deletedAt <- arbitrary

  pure student
    { deletedAt        = Just deletedAt
    , activeDataSource = Nothing
    }

Ad Hoc Overrides

The strategies discussed thus far assume we’re striving to re-use our generators. If we truly find we’re dealing with one-off cases it may be simpler to not even extract functions or define newtypes. For example, consider inlining our example generator.

  it "does xyz" $ do
    (student, deletedAt) <- generate arbitrary
    let deletedStudent =
      student
        { deletedAt=Just deletedAt
        , activeDataSource=Nothing
        }
    -- ...

This approach is excellent for triggering edge-cases. It has low overhead and is a solid way to start testing interesting combinations of state. We ought to be on the lookout, however, for opportunities to refactor ad hoc setup into newtypes or generator functions once we find that…

Balance

For large codebases, the formation of Arbitrary instances can have practical implications for

The effectiveness of an Arbitrary instance hinges on finding the balance between randomness and practical fixed example data. The approaches herein have helped Freckle engineers in finding this balance, yielding a highly maintainable test suite that’s been catching bugs and edge cases for over 5 years.