Custom reporting of VersionOne data in Clojure

VersionOne is an agile project management tool for planning and tracking epics, stories, themes, defects, tasks, tests and issues. This article is about a custom reporting application that consumes this data. I'll be using SCRUM terminology.

The code for these custom reports is available on github.


What it does

Before we dive into the implementation, let's take a quick tour of the custom reporting features.

Team Status

You can view the high level status of all teams on one page. There is a burndown of the todo hours, the cumulative flow of story status and a table of churn items which have been re-estimated, added to or removed from the sprint.


Story History

Clicking on the [hist] link opens a new tab in your browser showing the history of the story in chronological order, most recent first. The history shows only the changed values and displays the before and after. The VersionOne history forces you to click in to each change, look for fields with a + near them and expand them to identify changes. This custom view saves a lot of time as you can visually scan what people have applied what changes recently.


Retrospective

At the end of our sprints we meet to evaluate what worked well and what we can do better.
Burndowns, cumulative flows, velocity, stats (estimation vs actual, billable, capacity, efficiency, defects, tests, member contribution), failed review, churn, stories, splits, customer focus, you name it - you'll find it here!

Burndown comparison over the last 4 sprints helps identifying trends:


Churn shows how much work is being inserted and removed from the sprint:


High failed review counts can indicate a lack of communication between Dev/QA/Prod about what is required for the story:

You can view the other charts and tables at your leisure on the site.

Defect Rate

Per project you can see how many defects remain open, and the total number in the system.  The idea here is that as we near a release date, the open defects should be approaching zero. Inspired by chapter 16 of the Software project survival guide.


Open Items

Often several release projects are underway and it is useful to see a table of open stories to monitor the status of work remaining.


Roadmap

The roadmap shows the total story points per sprint. Dates in the past are done work, dates in the future are planned work. The table indicates:

  • whether projected work is reasonable
  • where effort is being focused
  • how cross team projects are lining up

The roadmap is presented as a pivot table which means you can view the data by team, project, customer, or any combination of those as aggregated data.


Fable

This page detects unbelievable stories. Fables are stories that have certain attributes missing, incorrect, or inconsistent. The page checks for:
  • a valid estimate
  • a success criteria
  • a description length greater than the template
  • 2 owners (Dev and QA)

Conclusion

VersionOne stores a lot of very useful data about what teams are doing. There are a plethora of interesting aspects to the data for reporting purposes which can be quite powerful when customized to your process interests.

If you are a VersionOne user, please give this reporting tool a try! Visit github for instructions on how to run it, and feel free to contact me with any questions.

In the next article, I will discuss how these reports are implemented in a Clojure webservice backend that applies transformations to VersionOne data API queries. The interface uses AngularJS databinding, Bootstrap layout and Google Visualization charts.

13 comments:

  1. I'm looking forward to your next post on the guts!

    ReplyDelete
  2. Hello.
    Is it possible to make this run on windows? If so, can you briefly tell me how?
    Thanks.

    ReplyDelete
  3. Hi Ferec,

    Yes you can run this on Windows (or any platform that has Java) very easily.
    (1) Clone or download the repository from https://github.com/timothypratley/vone.
    (2) Install the build tool https://github.com/technomancy/leiningen#windows.
    (3) Create a resources/vone.properties file containing:
    username = username
    password = password
    base-url = http://www3.v1host.com/MyCompany/VersionOne/rest-1.v1
    (4) start the webserver by running lein ring server. A browser will open pointing to http://localhost:3000 to display the reports.

    Please contact me if you have any difficulty, questions, feedback, or need pointers on customizing for your data.

    ReplyDelete
  4. Hi Ian,

    Thanks for your interest. I have posted more detail about the implementation here: http://timothypratley.blogspot.com/2014/02/custom-reporting-of-versionone-data-in_27.html.

    ReplyDelete
  5. Hi Timothy,

    I have followed your steps for setting up this on Windows 7 64 bit desktop. At step 4) running "lein ring server" am getting this error :

    ==============
    C:\Users\pajagann\Downloads\leiningen-master\bin>lein ring server

    Leiningen is missing its dependencies.
    Please run "lein bootstrap" in the leiningen-core/ directory
    with a stable release of Leiningen. See CONTRIBUTING.md for details.
    ===========

    Note: I have setup the Path environment variable pointing appropriately too.

    Any help is highly Appreciated.

    Thanks,
    -Partha

    ReplyDelete
  6. Hi Partha,

    Sounds like Leiningen is not installed quite right. Which method did you use to install it? Also did you try looking for leiningen-core and running "lein bootstrap" from it? I recommend trying the direct .bat download method described in https://github.com/technomancy/leiningen/blob/master/README.md

    "lein -version" should report 2.0 or higher before the project can be run.

    ReplyDelete
  7. Thanks Timothy for quick turnaround!

    My objective basically is to perform Custom reporting of VersionOne data for certain projects under it.

    Note: VersionOne URL of mine is accessible only with VPN

    I have fixed the lein part now as you suggested using batch script & version seems to be alright.
    Also I have created vone.properties under leiningen-master\resources after which when running "lein ring server" for the first time it asked me to run "lein bootstrap" from leiningen-core dir which completed fine and upon re-running "lein ring server" am landing into the below issue.

    Is it an issue in project.clj (or) am I missing something.

    =========
    C:\Users\pajagann>lein -version
    Leiningen 2.4.3-SNAPSHOT on Java 1.8.0_05 Java HotSpot(TM) 64-Bit Server VM

    C:\Users\pajagann>lein ring server
    'ring' is not a task. See 'lein help'.

    Did you mean this?
    run
    ==========

    I was expecting the report collected is displaying by browser getting launched with URL http://localhost:3000 as in your description.

    I am ok doing Teamviewer if required based on your convenience.

    -Partha

    ReplyDelete
  8. Hi Partha, that's great. Please run "lein ring server" in the vone directory that you cloned, not your home directory. The ring task is from a plugin defined in the project.clj in the vone root directory.

    ReplyDelete
  9. Sorry to trouble you again Timothy:( , am landing into http_proxy not set issue when running "lein ring server" though I have set http_proxy environment variable.

    ======
    C:\Users\pajagann\Documents\vone-master\vone-master>lein ring server

    (Could not transfer artifact com.google.appengine:appengine-local-runtime:pom:1.
    6.3.1 from/to releases (http://appengine-magic-mvn.googlecode.com/svn/releases/)
    : Connection to http://appengine-magic-mvn.googlecode.com refused)
    (Could not transfer artifact com.google.appengine:appengine-local-runtime-shared
    :pom:1.6.3.1 from/to releases (http://appengine-magic-mvn.googlecode.com/svn/rel
    eases/): Connection to http://appengine-magic-mvn.googlecode.com refused)
    (Could not transfer artifact com.google.appengine:appengine-tools-api:pom:1.6.3.
    1 from/to releases (http://appengine-magic-mvn.googlecode.com/svn/releases/): Co
    nnection to http://appengine-magic-mvn.googlecode.com refused)
    This could be due to a typo in :dependencies or network issues.
    If you are behind a proxy, try setting the 'http_proxy' environment variable.



    C:\Users\pajagann\Documents\vone-master\vone-master>echo %http_proxy%
    148.87.19.20:80

    ReplyDelete
  10. Hi Partha,

    Could you try removing that plugin? It is not necessary except to deploy to Google Apps which you don't need to do. To remove it, open the project.clj file and delete
    [appengine-magic "0.5.0"]

    Then execute
    lein ring server

    If other dependencies cannot be retrieved, then we need to figure out why Leiningen cannot download them.

    ReplyDelete
  11. That does the trick Timothy, it launched http://localhost:3000, however beyond that tried logging in with V1 Creds its not able to login or fetch some reports. Also I have vone.properties file updated under vone-master/resources dir too.

    Here's the exception am noticing :

    ============

    C:\Users\pajagann\Documents\vone-master\vone-master>lein ring server
    2014-06-24 13:56:48.983:INFO:oejs.Server:jetty-7.6.8.v20121106
    2014-06-24 13:56:49.033:INFO:oejs.AbstractConnector:Started SelectChannelConnect
    or@0.0.0.0:3000
    Started server on port 3000
    java.lang.IllegalArgumentException: Don't know how to create ISeq from: java.lan
    g.Long
    RT.java:505 clojure.lang.RT.seqFrom
    RT.java:486 clojure.lang.RT.seq
    RT.java:572 clojure.lang.RT.cons
    core.clj:29 clojure.core/cons
    core.clj:606 clojure.core/list*
    core.clj:619 clojure.core/apply
    services.clj:41 vone.services/index-sprints[fn]
    ArrayChunk.java:58 clojure.lang.ArrayChunk.reduce
    protocols.clj:98 clojure.core.protocols/fn
    protocols.clj:19 clojure.core.protocols/fn[fn]
    protocols.clj:31 clojure.core.protocols/seq-reduce
    protocols.clj:54 clojure.core.protocols/fn
    protocols.clj:13 clojure.core.protocols/fn[fn]
    core.clj:6177 clojure.core/reduce
    services.clj:40 vone.services/index-sprints
    services.clj:50 vone.services/team-sprints
    Var.java:411 clojure.lang.Var.invoke
    AFn.java:159 clojure.lang.AFn.applyToHelper
    Var.java:532 clojure.lang.Var.applyTo
    core.clj:617 clojure.core/apply
    private.clj:142 routegen.private/call
    core.clj:35 routegen.core/path-routes[fn]
    core.clj:94 compojure.core/make-route[fn]
    core.clj:40 compojure.core/if-route[fn]
    core.clj:25 compojure.core/if-method[fn]
    core.clj:107 compojure.core/routing[fn]
    core.clj:2443 clojure.core/some
    core.clj:107 compojure.core/routing
    RestFn.java:139 clojure.lang.RestFn.applyTo
    core.clj:619 clojure.core/apply
    core.clj:112 compojure.core/routes[fn]
    keyword_params.clj:32 ring.middleware.keyword-params/wrap-keyword-pa
    rams[fn]
    nested_params.clj:70 ring.middleware.nested-params/wrap-nested-para
    ms[fn]
    params.clj:58 ring.middleware.params/wrap-params[fn]
    app_servlet.clj:24 vone.app-servlet/with-401[fn]
    core.clj:107 compojure.core/routing[fn]
    core.clj:2443 clojure.core/some
    core.clj:107 compojure.core/routing
    RestFn.java:139 clojure.lang.RestFn.applyTo
    core.clj:619 clojure.core/apply
    core.clj:112 compojure.core/routes[fn]
    session.clj:96 noir.session/noir-session[fn]
    session.clj:85 ring.middleware.session/wrap-session[fn]
    Var.java:415 clojure.lang.Var.invoke
    reload.clj:18 ring.middleware.reload/wrap-reload[fn]
    stacktrace.clj:17 ring.middleware.stacktrace/wrap-stacktrace-log
    [fn]
    stacktrace.clj:80 ring.middleware.stacktrace/wrap-stacktrace-web
    [fn]

    ReplyDelete
  12. Detailed exception am getting in the backdrop running "lein ring server" are as follows :

    Note: my VersionOne URL uses https, is it the reason for the below exceptions ? and not seeing any data fetched under launched URL http://localhost:3000

    ============

    ava.util.concurrent.ExecutionException: java.net.SocketException: Software caus
    d connection abort: recv failed
    (Unknown Source) java.util.concurrent.FutureTask.report
    (Unknown Source) java.util.concurrent.FutureTask.get
    core.clj:2108 clojure.core/deref-future
    core.clj:6308 clojure.core/future-call[fn]
    core.clj:2128 clojure.core/deref
    core.clj:2487 clojure.core/map[fn]
    LazySeq.java:42 clojure.lang.LazySeq.sval
    LazySeq.java:60 clojure.lang.LazySeq.seq
    RT.java:484 clojure.lang.RT.seq
    core.clj:133 clojure.core/seq
    core.clj:2490 clojure.core/map[fn]
    LazySeq.java:42 clojure.lang.LazySeq.sval
    LazySeq.java:60 clojure.lang.LazySeq.seq
    RT.java:484 clojure.lang.RT.seq
    core.clj:133 clojure.core/seq
    core.clj:2479 clojure.core/map[fn]
    LazySeq.java:42 clojure.lang.LazySeq.sval
    LazySeq.java:60 clojure.lang.LazySeq.seq
    Cons.java:39 clojure.lang.Cons.next
    RT.java:598 clojure.lang.RT.next
    core.clj:64 clojure.core/next
    custom.clj:142 cheshire.custom/encode-seq
    custom.clj:30 cheshire.custom/eval4222[fn]
    custom.clj:175 cheshire.custom/encode-map
    custom.clj:30 cheshire.custom/eval4222[fn]
    custom.clj:175 cheshire.custom/encode-map
    custom.clj:30 cheshire.custom/eval4222[fn]
    custom.clj:46 cheshire.custom/encode*
    custom.clj:35 cheshire.custom/encode*
    private.clj:27 routegen.private/json
    private.clj:85 routegen.private/datasource
    Var.java:419 clojure.lang.Var.invoke
    private.clj:142 routegen.private/call
    core.clj:35 routegen.core/path-routes[fn]
    core.clj:94 compojure.core/make-route[fn]
    core.clj:40 compojure.core/if-route[fn]
    core.clj:25 compojure.core/if-method[fn]
    core.clj:107 compojure.core/routing[fn]
    core.clj:2443 clojure.core/some
    core.clj:107 compojure.core/routing
    RestFn.java:139 clojure.lang.RestFn.applyTo
    core.clj:619 clojure.core/apply
    core.clj:112 compojure.core/routes[fn]
    keyword_params.clj:32 ring.middleware.keyword-params/wrap-keyword-params[
    n]

    (Unknown Source) sun.net.www.http.HttpClient.parseHTTP
    (Unknown Source) sun.net.www.protocol.http.HttpURLConnection.doTunne
    ling
    (Unknown Source) sun.net.www.protocol.https.AbstractDelegateHttpsURL
    Connection.connect
    (Unknown Source) sun.net.www.protocol.https.HttpsURLConnectionImpl.c
    onnect
    core.clj:69 clj-http.lite.core/request
    Var.java:415 clojure.lang.Var.invoke
    client.clj:162 clj-http.lite.client/wrap-query-params[fn]
    client.clj:187 clj-http.lite.client/wrap-user-info[fn]
    =======================

    Thanks

    ReplyDelete
  13. Hi Partha,

    Thanks for persisting despite the difficulties. Those errors indicate that vone is unable to retrieve the Timeboxes (sprints) from VersionOne. This could be because:
    1) Vone is not using the proxy to make its requests
    2) Your data might not use timeboxes

    A proxy can be specified by setting the Java properties: .proxyHost and .proxyPort where is the client scheme used (normally 'http' or 'https'). To do this please edit the project.clj file just after the :profiles entry add this line:
    :jvm-opts ["-Dhttp.proxyHost=148.87.19.20"]

    If you are still having any troubles please email me at timothypratley@gmail.com

    One thing we could try is if you create a read only guest account on your VersionOne instance, I could test it out directly.

    ReplyDelete