Metrics as our compass not our destination

“In God we trust, everyone else bring data”-Mike Bloomberg

Metrics, reports, data, statistics: management is asking for them, not all teams are happy or interested in them, and the people who gather them spent an awful big amount of producing them and sometimes feels like a waste of time. So my question is, why bother? Why go in all the trouble? The answer is simple: because having the right data, you can improve and make better decisions.

People are very defensive and afraid when it comes to the metrics. We have been trained our whole lives into thinking that a bad grade is a reason for punishment and comes with some consequences, a bad performance review or a bad feedback from our manager is a reason for firing. Being accustomed to this for a very long time, we view metrics as the big bad wolf of our student, professional and personal life and it is totally understandable. However, this is where we are wrong. The truth is we should be viewing the metrics with a positive eye and use them for our benefit and grow.

By reviewing the data, we can understand, analyze and improve our system and our organization. In several discussions with my teams and management, we talk about some things that are not going well or some things that we do exceptionally well. The team has a perception, a common idea and understanding about these and this feeling is shared. I am sure you have all noticed it. It is these times, when everyone in the team points on the same problem or the best practice. And this is where the data come in, as I like to say, metrics are the representation and visualization of this perception. Whether this is a number/ratio or a graph, data can visually present the feelings/ perception of our teams, of our customers and help us understand “how bad” or “how well” things are.

That being said, it is important to make the following distinction: metrics, however helpful they are, should not become our purpose and should be not become our goal. As Simon Sinek mentioned in many of his speeches, in an infinite game, metrics show us how far we are from our goals and how fast or slow we are reaching our goals. So, in that sense, metrics are our compass on our journey to reach our destination. Many times, we fall in this trap though and we frequently ask questions or make statements like: “ we need to get better numbers” or “ the goal is to achieve 90% test coverage”. And why this statements are wrong? The answer is because in that way we are missing the bigger picture, we are missing the vision and our goal. In the example that was mentioned about the test coverage, the right to way to phrase this is that our goal is to increase the quality gates of our product. The important thing is to create a momentum, a somewhat linear and consistent improvement towards our goal and data will help us identify this momentum and improvement.

For the people who have worked with me, it is obvious that I am reporting freak. I do like collecting data and creating reports to visualize the momentum that we mentioned above and I have to admit that most of the times these reports helped significantly my teams, the clients that I have worked with and the management. However, I would like to focus a bit more on why our teams need data/metrics and present some of the metrics that could potentially help your teams as well to create this momentum and essentially reach the set goals (not the number, but the vision!).

Being an engineering myself, I spent a lot of doing calculus, statistics and learn to have squared approach to the problems. We can all agree that engineers are a brilliant and a particular group of people. In many occasions, mostly in retrospectives, it was hard to get the team to open up about the problems, improvements and action items. However, when we have created a more data oriented approach in the retros, we found ourselves discussing specific issues and actually mitigating those with actionable items, when and only when we pulled up some data.

So what kind of metrics would be appropriate to use?

The approach here could be the following: User Experience, DevOps and Team.

User Experience

Gathering customer feedback is a very crucial part of the Agile Mindset. We want to see how our changes are received by customer, which are the favorite and most common paths, which devices or browsers are most frequently use by our customers, and so on. Using App Insights/ google analytics and various other metrics in our products, we can gather data that will help us better understand our customer, their user journeys. These metrics can be very beneficial to the product teams, as well the Product Owner and will significantly help them with formulating MVP for their products and help prioritizing the Product Backlog.

DevOps

For the DevOps related metrics, one thing comes in mind: DORA metrics. DORA metrics consist of the following key measures:

· Deployment Frequency — How often an organization successfully releases to production

· Lead Time for Changes — The amount of time it takes a commit to get into production

· Change Failure Rate — The percentage of deployments causing a failure in production

· Time to Restore Service — How long it takes an organization to recover from a failure in production

Along with this metrics, other indicators that could potentially help on the technical aspect is counting the level of automation that our teams have reached. Automation (test automation or establishing CI/CD pipelines) will help with having a higher quality and more secure deployments/product releases. Example of metrics here could be test automation coverage (attention on the regression/smoke/sanity test suites, unit and integration testing).

Team

In regards to the team related data that we can gather, few examples can be:

1. Planned/Committed User Stories vs Done User Stories:

For this indicator, we need to review how many of the stories that were committed in the beginning of the Sprint were done by the end of the Sprint. You can do count both the number of stories and also you can do a Story Point version on the same ratio. This indicator will help the team plan better in the future. For example, if in a team we have 80% Sprint Goal completion ratio, and this is starting to become an overage, we see that the team is slightly overcommitting in the Sprint, and we can get a good predictable on what to expect in next iterations.

2. Sprint Goal Ratio

The Sprint Goals should be formulated this way that by the end of the Sprint, the team or the Product Owner can simply reply with a yes/no on whether or not the Goal is achieved. Having that in mind, we can count the ratio of the completed goals, the “yes”-s versus the “no”-s. Same as above, if the ratio seems to be having a lower trend multiple action items/ potential root causes can arise from the discussion with the team.

3. Scope Change

We all are aware that change is part of our daily routine. We should embrace the change, and be fast enough to adjust to it, but frequent changes have some downsides in our team’s dynamics. So for us, it is important to measure, the scope changes that occurs in a Sprint, so we could potentially identify action items. Frequent changes might be an indicator of the framework that we are using. For example, a team that is using Scrum on a 2 week iteration and has changes in the Sprint almost on daily basis, that could be an indicator that Kan Ban approach might be needed.

4. Type of work in the Sprint

From this indicator we can identify a lot of things for the state of our product. The measuring can be on Sprint basis and/or a Release base. Identify how much of the Sprint work is spent on new feature development, on fixing bugs, on customer support/onboarding are valuable information that could help drive some of our future decisions. For example, in the beginning of the onboarding of a new customer, it is expected that the support will be increased, as the customer familiarizes with the product, the support should gradually drop.

5. Tracking duration in Stages of Work.

Breaking down our systems into stages and count the progress of these stages individually, it is very important for us to identify inefficiency in our way of working. A practice that I have used before is monitor each stage of work. For example tracking the time for a user story from In Progress status to Review, tracking time for a PR to get approved and merged, if in your organization track separately the testing stages, tracking this time is essential to understand in which phase of “delivery” our team could make some improvements.

6. Track the estimations — Overestimations/ Underestimations

Tracking the estimations meaning whether or not we were close with our original assumption for complexity or the team that are using hour/day estimates, the original time estimation, can bring up to the table several aspects. By evaluating the estimations level, we can understand and discuss with the team, the clarity of the requirements in the user story, our knowledge in a specific technology or in a specific service/ product. The important thing to note here is that we should not mind how much the underestimation or overestimation was, the takeaway and action item for these kind of metrics, should to have a more clear idea of the complexity or the time in case we perform a similar activity in the future and approach it with more confidence the next time.

Some final thoughts, school is over, a bad grade or a lower ratio does not mean that we are doing a bad job, it just means that we probably have to work a bit differently and improve our current state. Metrics are indeed important, but they should not replace our vision/end goal, they are our compass, not our destination. For the teams that are interested in calculating some of these data/metrics, start simple, Scrum Masters and Program Managers engage your teams in the discussions and try to help them understand and read behind the number. Decrypting the data/metrics is important piece on the way to continuous improvement and creating a momentum.

An experienced project/program manger/Scrum Master working on enterprise projects.