We live in a world where time to market and delivery timelines are only getting shorter for technology transformation initiatives. Enterprise or ISV, the expectations are the same. Add to it the following additional needs –

  1. Higher Software Quality
  2. Faster Issue Identification & Resolution
  3. Better & Proactive Security
  4. Continuous Compliance
  5. Reduced Cost of Delivery & Operations

It does not get any tougher. 

It is both challenging and interesting time to be a Technology Leader; challenging because of the needs mentioned above; interesting because of the many technology advancements in the past decade some of which include –

  1. Public Cloud (aws, azure, google cloud)
  2. Microservices (kubernetes, docker, serverless, istio)
  3. AI/ML Frameworks (tensorflow, pytorch, mxnet)
  4. Analytics (snowflake, hadoop, tableau)
  5. API Frameworks (apigee, mulesoft)

Adopting DevOps practices is key to achieving the above stated goals using these advanced technologies. Defining, tracking and learning from the KPIs (on DevOps Dashboards) is critical to continuously improving.

KPIs can be classified across the following high level business goals

  1. Cost
  2. Time to Market
  3. Quality
  4. Security / Compliance
  5. Application Performance / Availability

It is important to first identify business priorities amongst the goals listed above. Identifying the top two priorities is more pragmatic.

Once the business goals are prioritized, the next step is to identify the detailed DevOps KPIs per business application. Some of the important DevOps KPIs are listed below –

KPIDescription
Lead TimeTotal elapsed time from feature identification to availability in production
Process TimeTotal time taken for actual work from feature identification to availability in production
%C/AWhat percentage of the time is the work delivered in this step acceptable to the next step in the value stream
Deployment FrequencyHow often deployments are performed
Defect DensityWhat percentage of the backlog is comprised of defects to be fixed
Mean Time to DetectionAverage time taken to identify an application issue
Mean Time to RecoverAverage time taken to recover from a failure
Mean Time Between FailuresAverage time between two failures
Automated Test Coverage% of test cases automated
Deployment TimeTime taken for each deployment
Change Failure Rate% of deployments that fail in production
Automated Tests Defect Rate% of automated tests that fail
Feature Usage% of target user base that is using a new feature
Batch SizeNumber of features (points) delivered per release
Defect RateFailure rate
Application AvailabilityAvailability of the application per month
Customer TicketsNumber of customer tickets per month
Defect Escape RateNumber of issues found in production as against lower environments

 

KPIs should also be identified for each of the stakeholder persona like the following –

  1. Product Owner
  2. Product Manager
  3. Scrum Master
  4. Technology Owner
  5. Engineering Manager
  6. QA Manager
  7. Developer
  8. Operations Manager
  9. Security Officer
  10. Compliance Officer
  11. Site Reliability Engineer

Some of the KPIs of importance for each of these roles are mentioned below –

RoleKPIs
Product Owner
  • Product Cost
  • Product Delivery Velocity
Product Manager
  • Feature Velocity
  • Product Quality
  • Feature Usage
  • Batch Size
Scrum Master
  • Lead Time
  • Epic / Sprint Efficiency
  • Sprint Batch Size
Technology Owner
  • Customer Tickets Rate
  • Application Availability
  • Defect Escape Rate
Engineering Manager
  • Process Time
  • Efficiency
  • Code Coverage
QA Manager
  • Automated Test Coverage
  • Defect Density
  • Defect Escape Rate
Developer
  • Deployment Time
  • Code Quality
Operations Manager
  • Mean Time to Recover
  • Mean Time Between Failures
  • Application Availability
Security Officer
  • Security Incidents
  • Security Vulnerabilities
  • Security Posture Score
Compliance Officer
  • OS Compliance Score
  • Process Compliance Score
  • System Compliance Score
Site Reliability Engineer
  • Production Incident Rate
  • Application Availability
  • Application Performance

The information and data points required to derive these KPIs can be found across the following type of tools and platforms –

  1. Cloud Provider Services
  2. ALM 
  3. ITSM 
  4. Continuous Integration
  5. Continuous Delivery / Deployment
  6. Code Quality 
  7. Security Testing
  8. Test Automation
  9. Compliance Assessment
  10. Code Repository
  11. Artifact Repository
  12. Monitoring
  13. Logging

Ability to gather information in real time, apply correlations, generate metrics and recommendations from the above tools and platforms is critical to a comprehensive DevOps Dashboard that can subsequently help drive continuous improvement.

You can either adopt an approach of building such comprehensive dashboards yourself or use platforms like Kaiburr that provide real time dashboards out of the box for all stakeholders with information from tools across the development life cycle.

Whatever be your approach, good luck with your technology transformation initiatives and I hope the information provided in this post gives you a good starting point. Cheers.