- on Wed 14 January 2015
Because your attention span. Stew O'Connor and I recently gave a talk on ScalaCheck, the property-based testing library for Scala. You can watch the video, or absorb it here in the Internet's Truest Form. Here are 29 GIFs you have to be a total ScalaCheck witch to get:
1. ScalaCheck is black magick...
2. ...that can make your tests VERY powerful.
The standard imperative way of writing unit tests is limited. Property-based tests are a functional approach to testing: instead of asserting how our code should behave in a specific situation, we define truths about the code's output based on the input.
3. You barely have to blink:
Scalacheck automatically generates many, many inputs and then automatically verifies properties you define about what your code should output for those inputs.
4. It's got a serious background:
The library takes inspiration and its name from Haskell's QuickCheck, and uses the types of your input data to infer how to generate examples. ScalaCheck is used by many major Scala open source projects like Akka and Play, and even by the Scala compiler itself.
5. Good things come in threes:
ScalaCheck generates input, takes these inputs and verifies properties on them, then if it finds failures shrinks the failure space to the smallest possible set of failing inputs. See all three happening in this simple example.
6. You get a lot of functionality right out of the bottle:
There are provided input generators for primitive types, Throwable, Date, tuples, functions, and many types of containers including Option, List, and Map.
7. To say nothing of your little helpers:
You can also make use of helper functions like
alphaNumChar, which fix our earlier example.
8. Dare to live on the edge:
It's important to cover edge cases in your automated tests. The provided generators have a bias towards generating edge case data specific to their type. For example, the Int generator will always generate
MIN_VALUE, 0 and 1 early on; the String generator tries Strings containing non-Roman characters first.
9. Sometimes it's time to get fancy:
10. It slices! It dices!
You can also compose generators using
.filter (also aliased as
.suchThat.) These three methods mean you can build generators using a for-yield. For a powerful example, check out this recursive JSON generator; for a sillier example, see here.
11. You've got to be careful with .suchThat...
.filter) generates all input and then filters out any that don't match the supplied condition. If your condition is too restrictive, you may end up with so many tests discarded that your tests don't run. Try
.retryUntil in that situation.
12. ...for more than one reason.
More philosophically, you want to be sure you’re enforcing the condition you're filtering your input data on somewhere. Your generator might assume all of your users' ages will be under 120, but witches can live to over 300 years old!
13. Sometimes the standard distribution isn't quite what you're looking for:
Prop.collect/classify to verify the actual distribution of your generated data and
Gen.frequency to tweak it to your heart's desire.
14. P1 bug: Beyoncé not found!
True story: we had a production bug slip through our ScalaCheck test suite because we were using
alphaNumString to generate test data and it never detected that our users couldn't find Beyoncé with the correct acute accent. Rather than fiddle with generators, the best solution for our dataset size was to have the generator select from our possible production queries, loaded in a text file. My coworker Will Fitzgerald applied this approach to a dictionary combined with usage statistics and
Gen.frequency to create a generator that spit out plausible English text, which turned out to be useful outside of tests. Thankfully, the
.sample method on generators extracts a plain value that can be used in any context you want.
15. Generators can do double duty for performance testing:
ScalaMeter, a microbenchmarking framework, uses a very similar DSL to generate values and you can easily convert generators to share between the two types of tests. Be careful of ScalaCheck's bias towards edge cases though!
16. There's a lot of ways to run:
You can use ScalaCheck's built-in runner to run assertion-based ScalaCheck tests from the command line.
17. It's easy to integrate:
18. Time isn't always on your side:
Cases where you can exhaust the entire test space are rare. Usually, you're limited by how long the tests will take to run. The concurrency level, number of successful tests required, and size of generated collections in ScalaCheck tests are all configurable.
19. Spin it right round:
If you're encoding & decoding, roundtrip properties are an elegant way to test both methods: take a generated input, encode it then decode the encoded value and verify that the result equals the original input.
20. Invalid inputs are just as important to test as valid ones:
The Basho team has a good writeup of their experience using property-based testing on negative inputs to discover and fix issues with the worker pool used for Riak.
21. Don't limit your powers:
22. You can command more than just methods:
ScalaCheck's input-output model is meant for testing functions, but the framework also provides a model for testing stateful code. After you specify possible commands with pre and post conditions, the tests generate sequences of commands and verify that the conditions have held. See it in action testing Redis here.
23. It only gets cooler from there:
Rickard Nilsson, the creator of ScalaCheck, uses the command model with Nix, the functional build tool, to test server networks. He generates VMs with randomly generated memory, OS versions, and IP addresses and uses commands to test what happens when they ping each other.
24. Property-based testing scales up:
Simulation testing applies the concept of property-based testing to entire systems. Simulant is a simulation testing framework written in Clojure; it uses statistical modelling to generate test data and records outputs. Properties become queries that can be verified strictly or fuzzily.
25. Even better, it scales down:
One of the most useful features of ScalaCheck is shrinking. When a failure is found, ScalaCheck stops generating new cases. It tries to apply the failing property against successively "smaller" failure cases so it can present you with the most specific information it can discover about the failure.
Shrink (and the concept of a "smaller" failure case) is defined per type--for example,
Int literally gets smaller. With a tuple it'll try to shrink each member individually, and it'll methodically remove chunks from a list.
26. In practice, you barely have to lift a finger:
Go watch Stew show ScalaCheck off in the video already!
27. Property-based testing: not only for Scala!
28. Thank you so much, you've been wonderful.
29. Bonus magick GIF:
I just think this one is cool.