Tests are great for making sure stuff works. But tests and documentation are often the last thing on a programmers mind. Even if you have a testing regime in place, implementing useful tests is actually quite an art that requires experience and thought.
Examples are a subset of tests.
Actually examples are a really important subset of tests because they inevitably are concerned with the API you are exposing, not the internals of your implementation. This is a very high payoff place to have tests. So much so you can think of examples as 'essential tests'. Not only does this mean they are important to provide, but it also means if you are providing examples then you are generating high value tests without additional deep thinking or experience required.
Examples make excellent documentation.
Examples are a really great way to learn anything new. This is especially true in programming. Including examples in documentation of code communicates clearly what the code does and how it should be used. They also get the reader thinking about scenarios where they could use this stuff in ways that reading a description often does not convey.
One thing people fear about documenting examples is 'what if they get out of date?'. Indeed this is a concern. But if our examples are really just important tests, then this concern disappears. Of course our examples will be kept up to date because we are using them as tests. Important tests that we need to pass. If we can achieve both testing and documentation from one activity the payoff to cost is much more appealing.
Another concern is if the examples are too far removed from the code then they wont be created as part of the coding process. We want our development cycle to be closely knit with producing examples so these don't become a separate activate which could fall by the wayside. Performing documentation as a separate task rarely works well unless you have someone dedicated on the case. Documentation generated from source code improves the chance of survival.
So here are my requirements for a useful example framework:
- Define examples in one place once
- Automatically part of test suite
- Automatically included in documentation
- Examples live in code
How might we achieve this in Clojure?
First I'd like to note that all the parts are already there, which is great! Clojure has a built in test framework and an autodoc project which supports usage documentation.
Capture examples in workflow
When developing a new function I always end up using it in some way and experimenting with it a little in a REPL. The first thing I do is write a function stub, then immediately after a very basic call to the function. From there I refine back and forward filling out the function, adding calls to it and fixing problems. Those experiments to just after the function definition then get turned into tests when I'm finished. Being able to do this is a really useful feature of Clojure. Clojure has an unfair advantage here being a functional language, it naturally lends itself to small discreet examples matching directly the function definitions.
What is missing?
- Unit tests do not automatically appear as usage in generated doc
- Unit tests are user named and not related to symbols or available for lookup
- Usage documentation is not tested, and is separate from the tests or code
- Usage documentation is not available at the REPL via (doc) or (doc-usage)
- Usage documentation coverage does not exist yet
- There is no support for multiple usage or tests per symbol
These issues can all be overcome and I'm volunteering my help to address them. I've made a start on how I think this should be tackled and am now looking for feedback on whether these are useful improvements to make and the right approach to take.
The code is on github.
The README describes what my goals are, what is remaining, and gives build instructions.
The central part is an example macro which can used very similarly to deftest. The difference being it is associated to a symbol for lookup later from autodoc or (doc-usage) and has a slightly more restrictive form to more suit output formatting.
I've implemented all the wikibook examples. Now they are in s-expressions I am confident I can easily adapt them to a new system very easily.
Lastly I want to re-iterate that Clojure already supports excellent test workflow and documentation. I've found it useful to myself to make some minor tweaks to achieve my goals. Now I'm at the (doc) lookup and autodoc generation part I want to more closely align with what Tom and Stuart are doing with the aim to help provide usage examples for Clojure itself.