yusheng-guo/kube-metrics-adapter
1
0
Fork 1
mirror of https://github.com/zalando-incubator/kube-metrics-adapter.git synced 2025-01-03 07:40:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #68 from zalando-incubator/skipper-collector-averagevalue

Browse Source 
Add support for averageValue for request-per-second Skipper metric
This commit is contained in:
Mikkel Oscar Lyderik Larsen
2019-10-10 08:09:30 +02:00
committed by GitHub
parent 0de5042d3d 76d2f74743
commit 2765ff9811
5 changed files with 125 additions and 39 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

58
README.md
View File

@ -1,10 +1,13 @@
# kube-metrics-adapter
[![Build Status](https://travis-ci.org/zalando-incubator/kube-metrics-adapter.svg?branch=master)](https://travis-ci.org/zalando-incubator/kube-metrics-adapter)
[![Coverage Status](https://coveralls.io/repos/github/zalando-incubator/kube-metrics-adapter/badge.svg?branch=master)](https://coveralls.io/github/zalando-incubator/kube-metrics-adapter?branch=master)
Kube Metrics Adapter is a general purpose metrics adapter for Kubernetes that
can collect and serve custom and external metrics for Horizontal Pod
Autoscaling.
It supports scaling based on [Prometheus metrics](https://prometheus.io/), [SQS queues](https://aws.amazon.com/sqs/) and others out of the box.
It discovers Horizontal Pod Autoscaling resources and starts to collect the
requested metrics and stores them in memory. It's implemented using the
[custom-metrics-apiserver](https://github.com/kubernetes-incubator/custom-metrics-apiserver)
@ -41,6 +44,18 @@ The `metric-config.*` annotations are used by the `kube-metrics-adapter` to
configure a collector for getting the metrics. In the above example it
configures a *json-path pod collector*.
## Kubernetes compatibility
Like the [support
policy](https://kubernetes.io/docs/setup/release/version-skew-policy/) offered
for Kubernetes, this project aims to support the latest three minor releases of
Kubernetes.
Currently the default supported API is `autoscaling/v2beta1`. However we aim to
move to `autoscaling/v2beta2` (available since `v1.12`) in the near future as
this adds a lot of improvements over `v2beta1`. The move to `v2beta2` will most
likely happen as soon as [GKE adds support for it](https://issuetracker.google.com/issues/135624588).
## Building
This project uses [Go modules](https://github.com/golang/go/wiki/Modules) as
@ -71,9 +86,9 @@ Currently only `json-path` collection is supported.
### Supported metrics
| Metric | Description | Type |
| ------------ | -------------- | ------- |
| *custom* | No predefined metrics. Metrics are generated from user defined queries. | Pods |
| Metric | Description | Type | K8s Versions |
| ------------ | -------------- | ------- | -- |
| *custom* | No predefined metrics. Metrics are generated from user defined queries. | Pods | `>=1.10` |
### Example
@ -151,10 +166,10 @@ the trade-offs between the two approaches.
### Supported metrics
| Metric | Description | Type | Kind |
| ------------ | -------------- | ------- | -- |
| `prometheus-query` | Generic metric which requires a user defined query. | External | |
| *custom* | No predefined metrics. Metrics are generated from user defined queries. | Object | *any* |
| Metric | Description | Type | Kind | K8s Versions |
| ------------ | -------------- | ------- | -- | -- |
| `prometheus-query` | Generic metric which requires a user defined query. | External | | `>=1.10` |
| *custom* | No predefined metrics. Metrics are generated from user defined queries. | Object | *any* | `>=1.10` |
### Example: External Metric
@ -259,9 +274,9 @@ box so users don't have to define those manually.
### Supported metrics
| Metric | Description | Type | Kind |
| ----------- | -------------- | ------ | ---- |
| `requests-per-second` | Scale based on requests per second for a certain ingress. | Object | `Ingress` |
| Metric | Description | Type | Kind | K8s Versions |
| ----------- | -------------- | ------ | ---- | ---- |
| `requests-per-second` | Scale based on requests per second for a certain ingress. | Object | `Ingress` | `>=1.14` (can work with `>=1.10`) |
### Example
@ -288,7 +303,10 @@ spec:
apiVersion: extensions/v1beta1
kind: Ingress
name: myapp
targetValue: 10 # this will be treated as targetAverageValue
averageValue: 10 # Only works with Kubernetes >=1.14
# for Kubernetes <1.14 you can use `targetValue` instead:
targetValue: 10 # this must be set, but has no effect if `averageValue` is defined.
# Otherwise it will be treated as targetAverageValue
```
### Metric weighting based on backend
@ -302,13 +320,17 @@ return the requests-per-second being sent to the `backend1`. The ingress annotat
the backend weights can be obtained can be specified through the flag `--skipper-backends-annotation`.
**Note:** As of Kubernetes v1.10 the HPA does not support `targetAverageValue` for
**Note:** For Kubernetes `<v1.14` the HPA does not support `averageValue` for
metrics of type `Object`. In case of requests per second it does not make sense
to scale on a summed value because you can not make the total requests per
second go down by adding more pods. For this reason the skipper collector will
automatically treat the value you define in `targetValue` as an average per pod
instead of a total sum.
**ONLY use `targetValue` if you are on Kubernetes
`<1.14`, it is not as percise as using `averageValue` and will not be supported
after Kubernetes `v1.16` is released according to the [support policy](https://kubernetes.io/docs/setup/release/version-skew-policy/).**
## AWS collector
The AWS collector allows scaling based on external metrics exposed by AWS
@ -340,9 +362,9 @@ PolicyDocument:
### Supported metrics
| Metric | Description | Type |
| ------------ | ------- | -- |
| `sqs-queue-length` | Scale based on SQS queue length | External |
| Metric | Description | Type | K8s Versions |
| ------------ | ------- | -- | -- |
| `sqs-queue-length` | Scale based on SQS queue length | External | `>=1.10` |
### Example
@ -388,9 +410,9 @@ The ZMON collector allows scaling based on external metrics exposed by
### Supported metrics
| Metric | Description | Type |
| ------------ | ------- | -- |
| `zmon-check` | Scale based on any ZMON check results | External |
| Metric | Description | Type | K8s Versions |
| ------------ | ------- | -- | -- |
| `zmon-check` | Scale based on any ZMON check results | External | `>=1.10` |
### Example

3
example/deploy/hpa.yaml
View File

@ -37,7 +37,8 @@ spec:
apiVersion: extensions/v1beta1
kind: Ingress
name: custom-metrics-consumer
targetValue: 10 # this will be treated as targetAverageValue
averageValue: 10
targetValue: 10 # this must be set, but has no effect if `averageValue` is defined.
- type: External
external:
metricName: sqs-queue-length

2
pkg/collector/collector.go
View File

@ -165,6 +165,7 @@ type MetricConfig struct {
ObjectReference custom_metrics.ObjectReference
PerReplica bool
Interval time.Duration
MetricSpec autoscalingv2.MetricSpec
}
// ParseHPAMetrics parses the HPA object into a list of metric configurations.
@ -206,6 +207,7 @@ func ParseHPAMetrics(hpa *autoscalingv2.HorizontalPodAutoscaler) ([]*MetricConfi
MetricTypeName: typeName,
ObjectReference: ref,
Config: map[string]string{},
MetricSpec: metric,
}
if metric.Type == autoscalingv2.ExternalMetricSourceType &&

9
pkg/collector/skipper_collector.go
View File

@ -72,8 +72,7 @@ type SkipperCollector struct {
}
// NewSkipperCollector initializes a new SkipperCollector.
func NewSkipperCollector(client kubernetes.Interface, plugin CollectorPlugin, hpa *autoscalingv2.HorizontalPodAutoscaler,
config *MetricConfig, interval time.Duration, backendAnnotations []string, backend string) (*SkipperCollector, error) {
func NewSkipperCollector(client kubernetes.Interface, plugin CollectorPlugin, hpa *autoscalingv2.HorizontalPodAutoscaler, config *MetricConfig, interval time.Duration, backendAnnotations []string, backend string) (*SkipperCollector, error) {
return &SkipperCollector{
client: client,
objectReference: config.ObjectReference,
@ -178,6 +177,10 @@ func (c *SkipperCollector) GetMetrics() ([]CollectedMetric, error) {
return nil, fmt.Errorf("expected to only get one metric value, got %d", len(values))
}
value := values[0]
// For Kubernetes <v1.14 we have to fall back to manual average
if c.config.MetricSpec.Object.Target.AverageValue == nil {
// get current replicas for the targeted scale object. This is used to
// calculate an average metric instead of total.
// targetAverageValue will be available in Kubernetes v1.12
@ -191,9 +194,9 @@ func (c *SkipperCollector) GetMetrics() ([]CollectedMetric, error) {
return nil, fmt.Errorf("unable to get average value for %d replicas", replicas)
}
value := values[0]
avgValue := float64(value.Custom.Value.MilliValue()) / float64(replicas)
value.Custom.Value = *resource.NewMilliQuantity(int64(avgValue), resource.DecimalSI)
}
return []CollectedMetric{value}, nil
}

64
pkg/collector/skipper_collector_test.go
View File

@ -107,6 +107,7 @@ func TestSkipperCollector(t *testing.T) {
ingressName string
collectedMetric int
expectError bool
fakedAverage bool
namespace string
backendWeights map[string]map[string]int
replicas int32
@ -140,6 +141,7 @@ func TestSkipperCollector(t *testing.T) {
metrics: []int{100, 1500, 700},
ingressName: "dummy-ingress",
collectedMetric: 150,
fakedAverage: true,
namespace: "default",
backend: "backend1",
backendWeights: map[string]map[string]int{testBackendWeightsAnnotation: {"backend2": 50, "backend1": 50}},
@ -147,6 +149,18 @@ func TestSkipperCollector(t *testing.T) {
readyReplicas: 5,
backendAnnotations: []string{testBackendWeightsAnnotation},
},
{
msg: "test multiple replicas not calculating average internally",
metrics: []int{100, 1500, 700},
ingressName: "dummy-ingress",
collectedMetric: 750, // 50% of 1500
namespace: "default",
backend: "backend1",
backendWeights: map[string]map[string]int{testBackendWeightsAnnotation: {"backend2": 50, "backend1": 50}},
replicas: 5, // this is not taken into account
readyReplicas: 5,
backendAnnotations: []string{testBackendWeightsAnnotation},
},
{
msg: "test zero weight backends",
metrics: []int{100, 1500, 700},
@ -164,6 +178,22 @@ func TestSkipperCollector(t *testing.T) {
metrics: []int{100, 1500, 700},
ingressName: "dummy-ingress",
collectedMetric: 300,
fakedAverage: true,
namespace: "default",
backend: "backend1",
backendWeights: map[string]map[string]int{
testBackendWeightsAnnotation: {"backend2": 20, "backend1": 80},
testStacksetWeightsAnnotation: {"backend2": 0, "backend1": 100},
},
replicas: 5,
readyReplicas: 5,
backendAnnotations: []string{testBackendWeightsAnnotation, testStacksetWeightsAnnotation},
},
{
msg: "test multiple backend annotation not calculating average internally",
metrics: []int{100, 1500, 700},
ingressName: "dummy-ingress",
collectedMetric: 1500,
namespace: "default",
backend: "backend1",
backendWeights: map[string]map[string]int{
@ -227,6 +257,22 @@ func TestSkipperCollector(t *testing.T) {
metrics: []int{100, 1500, 700},
ingressName: "dummy-ingress",
collectedMetric: 60,
fakedAverage: true,
namespace: "default",
backend: "backend2",
backendWeights: map[string]map[string]int{
testBackendWeightsAnnotation: {"backend2": 20, "backend1": 80},
testStacksetWeightsAnnotation: {"backend1": 100},
},
replicas: 5,
readyReplicas: 5,
backendAnnotations: []string{testBackendWeightsAnnotation, testStacksetWeightsAnnotation},
},
{
msg: "test partial backend annotations not calculating average internally",
metrics: []int{100, 1500, 700},
ingressName: "dummy-ingress",
collectedMetric: 300,
namespace: "default",
backend: "backend2",
backendWeights: map[string]map[string]int{
@ -244,7 +290,7 @@ func TestSkipperCollector(t *testing.T) {
require.NoError(t, err)
plugin := makePlugin(tc.metrics)
hpa := makeHPA(tc.ingressName, tc.backend)
config := makeConfig(tc.backend)
config := makeConfig(tc.backend, tc.fakedAverage)
_, err = newDeployment(client, tc.namespace, tc.backend, tc.replicas, tc.readyReplicas)
require.NoError(t, err)
collector, err := NewSkipperCollector(client, plugin, hpa, config, time.Minute, tc.backendAnnotations, tc.backend)
@ -314,10 +360,22 @@ func makeHPA(ingressName, backend string) *autoscalingv2.HorizontalPodAutoscaler
},
}
}
func makeConfig(backend string) *MetricConfig {
return &MetricConfig{
func makeConfig(backend string, fakedAverage bool) *MetricConfig {
config := &MetricConfig{
MetricTypeName: MetricTypeName{Metric: autoscalingv2.MetricIdentifier{Name: fmt.Sprintf("%s,%s", rpsMetricName, backend)}},
MetricSpec: autoscalingv2.MetricSpec{
Object: &autoscalingv2.ObjectMetricSource{
Target: autoscalingv2.MetricTarget{},
},
},
}
if fakedAverage {
config.MetricSpec.Object.Target.Value = resource.NewQuantity(10, resource.DecimalSI)
} else {
config.MetricSpec.Object.Target.AverageValue = resource.NewQuantity(10, resource.DecimalSI)
}
return config
}
type FakeCollectorPlugin struct {