“They’re called KPI and not KPT because they are only key performance indicators, not truths.”
Bjarte Bogsnes at #BizAgility2017
Let’s start with this question: to what extent can the data-driven approach to project management also be applied to Agile project management? Is it just a matter of metrics and KPIs, presumably different, or is there more to it?
Traditional Project Management is “deterministic”. First you define exactly what you want to achieve, then you derive the necessary times and costs, and make in advance “the film” of all the activities that you have to perform (aka Project Management Plan). “Success” consists of releasing the originally planned deliverables at the end of the project, deviating as little as possible from the work plan and staying within budget.
On the other hand, Agile Project Management is “relativistic”. First you define the essential characteristics of what you want to realize (aka Minimum Viable Product) with set times and budgets, but foregoing capturing all the details. Then, rather than the film, you write a rough screenplay of the project (aka Product Roadmap). “Success” consists of releasing the greatest value of business value in the shortest possible time, through work iterations and incremental construction of the result.
This completely changes the management and control philosophy. The “turnkey” project, where everything is established a priori and client/user engagement is rarefied, becomes a true partnership between the product owner (client/business) and product creator (team), with daily engagement.
Requirements and characteristics are defined together with the client, and every 2-3 weeks increases in functionality are released into production, which ensures an early return of business. The fact that all technical and business stakeholders are constantly together “on the case” allows you to manage change as the norm and not as the exception, agreeing on elimination of requirements that are not very useful along the way and replacing them with more current ones.
Metrics for deterministic projects and metrics for agile projects
The preferred metrics in deterministic projects are those of so-called “predictability”, with which you can measure the deviations between the actual performance of the project and the planned performance (baseline). Above all, those provided for by the Earned Value method, with which to periodically evaluate the “real” progress of deliverables and actual costs incurred.
In a nutshell, the classical view is project-centered; you have to complete the project in compliance with at least the triple constraint (scope, time, cost). The Agile vision on the other hand is product-centered, with a unique basic philosophy: a project is not simply completed but realized and a living product in constant evolution is released.
The focus on the product widens the spectrum of measures that you can use. In a software development project, for example, the metrics of interest are not only those derivable from classic project management tracking tools. Data are everywhere, in different components and systems, and from the outset there is no unified view.
The following are some of the metrics that can be obtained from Agile Project Tracking systems like Jira, used for issue management (e.g. task, bug, etc.) and task boards:
- Total done: total number of completed tasks. Indicates the volume of work performed.
- Velocity: user stories/tasks implemented by the team in a work iteration. Indicates team productivity.
- Bug: number of bugs counted. Indicates the quality level and need for code factorizations.
- Tag: used to label two different types of task: factoring tasks and new functionalities, in such a way as to keep effort ratings separate for the two aspects.
- Recidivism: the rate of backward movement of a task card on the task board. For example, with reference to the task board of the following figure, a task card moved to the “Done” column that returns to the “In Progress” column following detection of a bug. If B is the number of backward movements and F the number of forward movements of the card, recidivism is expressed in percentage terms as B/(F+B)x100.
From the Source Control Systems (e.g. GitHub, Bitbucket, Mercurial, …) that you use for collaborative management and code versioning, you can draw indicators that allow you to answer questions such as:
- Who is working on what?
- How effectively are team members working together?
- How accurate were the estimates?
- How accurately were tasks sized?
- How really productive is the team?
From the Continuous Integration (CI) and deployment tools (e.g. Jenkins, …), with which you make the build, run automatic and regression tests, produce end artifacts and enable the preparation of the code on different environments, you can derive metrics that allow you to answer questions such as:
- How quickly are you releasing modifications to the client/user and how fast could you instead do so?
- With what level of consistency are you releasing results?
- Are you producing a code with quality?
Continuous data-driven
The data-driven approach in agile projects is not, however, differentiated only in the various measures considered (metric product-oriented vs. metric project-oriented) but also, and especially, in the frequency of information collection.
In classical management, data collection to determine project status is carried out on a regular basis, according to the so-called period of reporting (distance between 2 consecutive WPSs – work progress statuses) The emphasis is on determining the deviations on the date of actual performance from those planned, in order to take any corrective action and try to put the project back on the baseline “tracks”.
On the other hand, in Agile management the password is “continuous“. Continuous integration, continuous delivery, continuous improvement, continuous testing, continuous-whatever-you-want. In other words, the iterative organization of work and the incremental construction of the result characterize a whole range of elements in a “continuous” or “quasi-continuous” way, from planning to monitoring and control of progress, from moments of dialogue and communication to the collection and application of lessons learned.
In an Agile project, you spend an even greater planning effort, in absolute terms, than that used in a traditional project. However, it is camouflaged by “dilution” over different work iterations, instead of being concentrated in the initial phases of the project.
This principle also applies to the cycle of collection and analysis of data to support decisions, which is carried out on two loops grafted into one another.
Data collection, analysis and possible adjustment actions are performed and discussed by the agile team daily (Daily Stand-up Meeting) and at the end of each work iteration (Review and Retrospective Meeting). The actions are restricted to a more limited scope (aka, only tasks processed during the iteration) but with a much higher “sampling rate”, which is what ultimately determines the true “agile” nature of this type of approach.
Conclusions
By focusing on delivering a product/result in constant evolution rather than on the completion of works in accordance with the triple “scope-time-cost” constraint, Agile project management is inherently data-driven. Alongside the traditional metrics of predictability (e.g. Earned Value) typical of “deterministic” project management, there are also those derived from the different components and systems used in the making of the product. In the case of software projects, KPIs of interest are derived from code versioning systems and from the infrastructure of continuous integration and application deployment.
The advantage of having a potentially greater number of measurements, above all carried out with greater frequency both at individual work iteration level and on a daily basis, is partly balanced by the fact that the data are everywhere, in different components and systems, and that from the outset there is no single, unified view. Are you ready to build it in your next agile project?
Marco Caressa
Recommended reading for learning more
Borror, C. (2009). “The Define Measure Analyze Improve Control (DMAIC) Process.” Retrieved February 14, 2015, from http://asq.org/learn-about-quality/six-sigma/overview/dmaic.html
Ghera, B. (2011). “Project and Program Management Analytics.” Retrieved February 10, 2015, from http://www.pmi.org/~/media/PDF/Knowledge-Shelf/Gera_2011(2).ashx
Mavenlink. (2013). “Using Analytics for Project Management.” Retrieved February 11, 2015, from http://blog.mavenlink.com/using-analytics-for-project-management
Project Management Institute (2014). A Guide to the Project Management Body of Knowledge (PMBOK® Guide), 5th ed. Newton Square, Pennsylvania: Project Management Institute (PMI).
C. W. H. Davis. “Agile Metrics in Action. How to measure and improve team performance” 2015, ed. Manning
