Metrics

Introduction

The Agreement conformed from several Guarantees contain all the information needed to compute them. Each guarantee contains an objective, which can involve one or more metrics. The metric is the instrument used to fetch the required information and extract a value for the system to verify the fulfilment of the objective.

Collectors

The collectors are the component which receiving a metric as input, are able to connect to the different data sources to extract the information needed for it, and compute the final result.

Actually, there are 3 different collectors available. The metrics are modeled as a JSON object following the iAgree specification. Each collector has his own syntax and they are similar in order to be consistent but there are some differences in order to extend the features.

Event Collector

The Event Collector is capable of using different data sources as input and join the information in order to correlate different payloads composing more complex metrics. Actually it can fetch information from Github, Pivotal Tracker, Travis CI, CodeClimate and Heroku.

Patterns

The metrics available to use in the Event Collector follow 4 different patterns:

1. Number of [Event] in [Tool] every [Period] by [TEAM|MEMBER]
2. [MAX|MIN|AVG|STD|NEWEST|LATEST] [Property] value of [Event] in [Tool] every [Period] by team.
3. [Frequency] distribution of [Event] in [Tool] every [Period] by team.
4. Percentage of [Event1] in [Tool1] correlated with [Event2] in [Tool2] within [window] every [Period] by team.

Right now these are the events available for each tool:

Here we are showing 4 different examples with their corresponding JSON format using the patterns above. Bear in mind that this is the DSL of the metric and the period is not displayed because it belongs to the guarantee:

1. Number of STARTED_STORIES in PIVOTAL every DAY by MEMBER

NUMBER_PT_STARTEDSTORIES
{
    "metric": {
      "computing": "actual",
      "element": "number",
      "event": {
        "pivotal": {
          "activity": {
            "highlight": "finished"
          }
        }
      },
      "scope": {
        "$ref": "#/context/definitions/scopes/development"
      }
    }
}

2. AVG COVERAGE value of COVERAGE_REPORT in CODECLIMATE every WEEK by team.

VALUE_CC_COVERAGE_AVG
{
    "metric": {
      "computing": "actual",
      "element": {
        "value": {
          "parameter": "attributes.covered_percent",
          "return": "avg",
          "traceback": true
        }
      },
      "event": {
        "codeclimate": {
          "coverage": {}
        }
      },
      "scope": {
        "$ref": "#/context/definitions/scopes/development"
      }
    }

3. DAILY distribution of SUCCESSFUL_BUILDS in TRAVIS every WEEK by team

STDEV_TR_SUCCESSFULBUILDS_DAILY
{
    "metric": {
      "computing": "string",
      "element": {
        "stdev": {
          "period": "daily"
        }
      },
      "event": {
        "travis": {
          "builds_public": {
            "@type": "build",
            "state": "passed"          
          }
        }
      },
      "scope": {
        "$ref": "#/context/definitions/scopes/development"
      }
    }

4. Percentage of NEW_BRANCH in GITHUB correlated with START_STORY in PIVOTAL_TRACKER within 1_HOUR every WEEK by team.

PERCENTAGE_GH_NEWBRANCH_PT_STARTEDSTORIES
{
    "metric": {
      "computing": "actual",
      "element": {
        "percentage": {
          "related": {
            "github": {
              "events": {
                "type": "CreateEvent",
                "payload": {
                  "ref_type": "branch"
                }
              }
            },
            "window": 3600
          }
        }
      },
      "event": {
        "pivotal": {
          "activity": {
            "highlight": "started"
          }
        }
      },
      "scope": {
        "$ref": "#/context/definitions/scopes/development"
      }
    }

For a full list of metric pattern examples, please follow this link where you can found the following examples:

Non correlation metrics:

Correlation metrics:

Pivotal Tracker Collector

The PT Collector connects with Pivotal Tracker and can produce more specific metrics using the different payloads.

Examples

There is a webpage with available examples for public usage.

GitHub Collector

Using GitHub API v4, it can access to the information from GitHub Projects Kanbans in order to keep track of the different cards and the changes.

Examples

There is a webpage with available examples for public usage.