Skip to content

Commit

Permalink
feat: Provide scrapers for Azure CDN, Cache for Redis Enterprise, Dat…
Browse files Browse the repository at this point in the history
…abase for MariaDB, Load Balancer & Virtual Network (#1883)

* Provide configuration

Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>

* Add serialization tests

Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>

* Add resource discovery

Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>

* Add validation

Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>

* Provide scrapers

Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>

* Fix local setup & bug in VNET

Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>

* Provide changelog entries

Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>

* Provide documentation

Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>

* Add missing description and leave out Atlassian for local testing

Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>

* Provide note on CDN metrics

Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
  • Loading branch information
tomkerkhove authored Dec 17, 2021
1 parent a5b19ac commit 9ee0930
Show file tree
Hide file tree
Showing 60 changed files with 2,280 additions and 256 deletions.
20 changes: 20 additions & 0 deletions changelog/content/experimental/unreleased.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,16 @@ version:

- {{% tag added %}} Provide scraper for Azure Application Insights ([docs](https://docs.promitor.io/configuration/v2.x/metrics/application-insights)
| [#1239](https://github.com/tomkerkhove/promitor/issues/1239))
- {{% tag added %}} Provide scraper for Azure Cache for Redis Enterprise ([docs](https://docs.promitor.io/configuration/v2.x/metrics/redis-enterprise-cache)
| [#1880](https://github.com/tomkerkhove/promitor/issues/1880))
- {{% tag added %}} Provide scraper for Azure CDN ([docs](https://docs.promitor.io/configuration/v2.x/metrics/cdn)
| [#1664](https://github.com/tomkerkhove/promitor/issues/1664))
- {{% tag added %}} Provide scraper for Azure Database for MariaDB ([docs](https://docs.promitor.io/configuration/v2.x/metrics/maria-db)
| [#1795](https://github.com/tomkerkhove/promitor/issues/1795))
- {{% tag added %}} Provide scraper for Azure Load Balancer ([docs](https://docs.promitor.io/configuration/v2.x/metrics/load-balancer)
| [#1845](https://github.com/tomkerkhove/promitor/issues/1845))
- {{% tag added %}} Provide scraper for Azure Virtual Network ([docs](https://docs.promitor.io/configuration/v2.x/metrics/virtual-network)
| [#1170](https://github.com/tomkerkhove/promitor/issues/1170))
- {{% tag added %}} Provide support for scraping Azure Database for PostgreSQL Flexible & Hyperscale resources ([docs](https://docs.promitor.io/configuration/v2.x/metrics/postgresql)
| [#1870](https://github.com/tomkerkhove/promitor/issues/1870))
- {{% tag added %}} Information concerning .NET & OS in logs on startup
Expand All @@ -19,6 +29,16 @@ version:

- {{% tag added %}} Provide scraper for Azure Application Insights ([docs](https://docs.promitor.io/configuration/v2.x/metrics/application-insights)
| [#1239](https://github.com/tomkerkhove/promitor/issues/1239))
- {{% tag added %}} Provide scraper for Azure Cache for Redis Enterprise ([docs](https://docs.promitor.io/configuration/v2.x/metrics/redis-enterprise-cache)
| [#1880](https://github.com/tomkerkhove/promitor/issues/1880))
- {{% tag added %}} Provide scraper for Azure CDN ([docs](https://docs.promitor.io/configuration/v2.x/metrics/cdn)
| [#1664](https://github.com/tomkerkhove/promitor/issues/1664))
- {{% tag added %}} Provide scraper for Azure Database for MariaDB ([docs](https://docs.promitor.io/configuration/v2.x/metrics/maria-db)
| [#1795](https://github.com/tomkerkhove/promitor/issues/1795))
- {{% tag added %}} Provide scraper for Azure Load Balancer ([docs](https://docs.promitor.io/configuration/v2.x/metrics/load-balancer)
| [#1845](https://github.com/tomkerkhove/promitor/issues/1845))
- {{% tag added %}} Provide scraper for Azure Virtual Network ([docs](https://docs.promitor.io/configuration/v2.x/metrics/virtual-network)
| [#1170](https://github.com/tomkerkhove/promitor/issues/1170))
- {{% tag added %}} Provide support for scraping Azure Database for PostgreSQL Flexible & Hyperscale resources ([docs](https://docs.promitor.io/configuration/v2.x/metrics/postgresql)
| [#1870](https://github.com/tomkerkhove/promitor/issues/1870))
- {{% tag added %}} Information concerning .NET & OS in logs on startup
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ resourceDiscoveryGroups:
type: AppPlan
- name: automation-accounts
type: AutomationAccount
- name: cdn-landscape
type: Cdn
- name: container-instances
type: ContainerInstance
- name: container-registry-landscape
Expand All @@ -35,14 +37,20 @@ resourceDiscoveryGroups:
type: IoTHub
- name: key-vaults
type: KeyVault
- name: load-balancers
type: LoadBalancer
- name: autoscaling-rules
type: MonitorAutoscale
- name: mariadb-servers
type: MariaDb
- name: network-interfaces
type: NetworkInterface
- name: postgres-databases
type: PostgreSql
- name: redis-caches
type: RedisCache
- name: redis-enterprise-caches
type: RedisEnterpriseCache
- name: sql-databases
type: SqlDatabase
- name: sql-managed-instances
Expand All @@ -63,6 +71,8 @@ resourceDiscoveryGroups:
type: VirtualMachine
- name: virtual-machine-scale-sets
type: VirtualMachineScaleSet
- name: virtual-networks
type: VirtualNetwork
- name: web-apps-north-europe
type: WebApp
criteria:
Expand Down
36 changes: 36 additions & 0 deletions config/promitor/scraper/metrics.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,42 @@ metrics:
# Application Insights with data in Log Analytics
- name: promitor-automation-data-generation-we-telemetry
resourceGroupName: promitor-automation-data-generation
- name: promitor_demo_cdn_requests_discovered
description: "Amount of requests sent to Azure CDN"
resourceType: Cdn
azureMetricConfiguration:
metricName: RequestCount
aggregation:
type: Total
resourceDiscoveryGroups:
- name: cdn-landscape
- name: promitor_demo_maria_db_cpu_discovered
description: "CPU usage of Maria DB servers"
resourceType: MariaDb
azureMetricConfiguration:
metricName: cpu_percent
aggregation:
type: Average
resourceDiscoveryGroups:
- name: mariadb-servers
- name: promitor_demo_load_balancers_requests_discovered
description: "Amount of bytes sent through the Azure Load Balancer"
resourceType: LoadBalancer
azureMetricConfiguration:
metricName: ByteCount
aggregation:
type: Total
resourceDiscoveryGroups:
- name: load-balancers
- name: promitor_demo_vnet_ddos_discovered
description: "Amount of requests at the origin of our Azure CDN"
resourceType: VirtualNetwork
azureMetricConfiguration:
metricName: IfUnderDDoSAttack
aggregation:
type: Maximum
resourceDiscoveryGroups:
- name: virtual-networks
# This uses our large-scale data set containing 1000+ Azure Logic App instances
# Uncomment if you want to test with this scale
# - name: azure_logic_apps_failed_run_discovery
Expand Down
10 changes: 5 additions & 5 deletions config/promitor/scraper/runtime.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,11 @@ metricSinks:
host: graphite
port: 8125
metricPrefix: promitor.
atlassianStatuspage:
pageId: 4mwc0ny6bgw1
systemMetricMapping:
- id: 333802sjqpr9
promitorMetricName: promitor_demo_documentation_availability
# atlassianStatuspage:
# pageId: 4mwc0ny6bgw1
# systemMetricMapping:
# - id: 333802sjqpr9
# promitorMetricName: promitor_demo_documentation_availability
metricsConfiguration:
absolutePath: /config/metrics-declaration.yaml
telemetry:
Expand Down
41 changes: 41 additions & 0 deletions docs/configuration/v2.x/metrics/cdn.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
layout: default
title: Azure Content Delivery Network (CDN) Declaration
---

## Azure Content Delivery Network (CDN)

![Availability Badge](https://img.shields.io/badge/Available%20Starting-v2.6-green.svg)![Resource Discovery Support Badge](https://img.shields.io/badge/Support%20for%20Resource%20Discovery-Yes-green.svg)

You can declare to scrape an Azure CDN via the `Cdn` resource
type.

When using declared resources, the following fields need to be provided:

- `cdnName` - The name of the Azure CDN resource

All supported metrics are documented in the official [Azure Monitor documentation](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported#microsoftcdnprofiles).

> 🚨 The availability of metrics depends on the SKU of the Azure CDN resource.
Example:

```yaml
name: azure_cdn_requests
description: "Amount of requests sent to Azure CDN"
resourceType: Cdn
azureMetricConfiguration:
metricName: RequestCount
aggregation:
type: Total
resources: # Optional, required when no resource discovery is configured
- cdnName: promitor-cdn-1
- cdnName: promitor-cdn-2
resourceDiscoveryGroups: # Optional, requires Promitor Resource Discovery agent (https://promitor.io/concepts/how-it-works#using-resource-discovery)
- name: cdn-landscape
```
<!-- markdownlint-disable MD033 -->
[&larr; back to metrics declarations](/configuration/v2.x/metrics)<br />
[&larr; back to introduction](/)
<!-- markdownlint-enable -->
5 changes: 5 additions & 0 deletions docs/configuration/v2.x/metrics/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,15 @@ We also provide a simplified way to scrape the following Azure resources:
- [Azure Application Insights](application-insights)
- [Azure App Plan](app-plan)
- [Azure Cache for Redis](redis-cache)
- [Azure Cache for Redis Enterprise](redis-enterprise-cache)
- [Azure Content Delivery Network (CDN)](cdn)
- [Azure Container Instances](container-instances)
- [Azure Container Registry](container-registry)
- [Azure Cosmos DB](cosmos-db)
- [Azure Data Factory](data-factory)
- [Azure Data Share](data-share)
- [Azure Database for PostgreSQL](postgresql)
- [Azure Database for MariaDB](maria-db)
- [Azure Event Hubs](event-hubs)
- [Azure Express Route Circuit](express-route-circuit)
- [Azure Front Door](front-door)
Expand All @@ -37,6 +40,7 @@ We also provide a simplified way to scrape the following Azure resources:
- [Azure IoT Hub Device Provisioning Service (DPS)](iot-hub-device-provisioning-service)
- [Azure Key Vault](key-vault)
- [Azure Kubernetes Service](kubernetes)
- [Azure Load Balancer](load-balancer)
- [Azure Logic Apps](logic-apps)
- [Azure Monitor Autoscale](monitor-autoscale)
- [Azure Network Gateway](network-gateway)
Expand All @@ -55,6 +59,7 @@ We also provide a simplified way to scrape the following Azure resources:
- [Azure Synapse (Workspace)](synapse-workspace)
- [Azure Virtual Machine](virtual-machine)
- [Azure Virtual Machine Scale Set (VMSS)](virtual-machine-scale-set)
- [Azure Virtual Network](virtual-network)
- [Azure Web App](web-app)

Want to help out? Create an issue and [contribute a new scraper](https://github.com/tomkerkhove/promitor/blob/master/adding-a-new-scraper.md).
Expand Down
39 changes: 39 additions & 0 deletions docs/configuration/v2.x/metrics/load-balancer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
layout: default
title: Azure Load Balancer Declaration
---

## Azure Load Balancer

![Availability Badge](https://img.shields.io/badge/Available%20Starting-v2.6-green.svg)![Resource Discovery Support Badge](https://img.shields.io/badge/Support%20for%20Resource%20Discovery-Yes-green.svg)

You can declare to scrape an Azure Load Balancer via the `LoadBalancer` resource
type.

When using declared resources, the following fields need to be provided:

- `loadBalancerName` - The name of the Azure Load Balancer resource

All supported metrics are documented in the official [Azure Monitor documentation](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported#microsoftnetworkloadbalancers).

Example:

```yaml
name: azure_load_balancer_traffic_bytes
description: "Average amount of bytes sent through an Azure Load Balancer"
resourceType: LoadBalancer
azureMetricConfiguration:
metricName: ByteCount
aggregation:
type: Average
resources: # Optional, required when no resource discovery is configured
- loadBalancerName: promitor-load-balancer-1
- loadBalancerName: promitor-load-balancer-2
resourceDiscoveryGroups: # Optional, requires Promitor Resource Discovery agent (https://promitor.io/concepts/how-it-works#using-resource-discovery)
- name: load-balancer-landscape
```
<!-- markdownlint-disable MD033 -->
[&larr; back to metrics declarations](/configuration/v2.x/metrics)<br />
[&larr; back to introduction](/)
<!-- markdownlint-enable -->
39 changes: 39 additions & 0 deletions docs/configuration/v2.x/metrics/maria-db.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
layout: default
title: Azure Database for MariaDB Declaration
---

## Azure Database for MariaDB

![Availability Badge](https://img.shields.io/badge/Available%20Starting-v2.6-green.svg)![Resource Discovery Support Badge](https://img.shields.io/badge/Support%20for%20Resource%20Discovery-Yes-green.svg)

You can declare to scrape an Azure Database for MariaDB via the `MariaDb` resource
type.

When using declared resources, the following fields need to be provided:

- `serverName` - The name of the Azure Database for MariaDB server

All supported metrics are documented in the official [Azure Monitor documentation](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported#microsoftdbformariadbservers).

Example:

```yaml
name: azure_db_mariadb_percentage_cpu
description: "Average percentage cpu usage on an Azure Database for MariaDB"
resourceType: MariaDb
azureMetricConfiguration:
metricName: cpu_percent
aggregation:
type: Average
resources: # Optional, required when no resource discovery is configured
- serverName: promitor-maria-db-1
- serverName: promitor-maria-db-2
resourceDiscoveryGroups: # Optional, requires Promitor Resource Discovery agent (https://promitor.io/concepts/how-it-works#using-resource-discovery)
- name: maria-db-landscape
```
<!-- markdownlint-disable MD033 -->
[&larr; back to metrics declarations](/configuration/v2.x/metrics)<br />
[&larr; back to introduction](/)
<!-- markdownlint-enable -->
39 changes: 39 additions & 0 deletions docs/configuration/v2.x/metrics/redis-enterprise-cache.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
layout: default
title: Azure Cache for Redis Enterprise Declaration
---

## Azure Cache for Redis Enterprise

![Availability Badge](https://img.shields.io/badge/Available%20Starting-v2.6-green.svg)![Resource Discovery Support Badge](https://img.shields.io/badge/Support%20for%20Resource%20Discovery-Yes-green.svg)

You can declare to scrape an Azure Cache for Redis Enterprise via the `RedisEnterpriseCache` resource
type.

When using declared resources, the following fields need to be provided:

- `cacheName` - The name of the Azure Cache for Redis Enterprise resource

All supported metrics are documented in the official [Azure Monitor documentation](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported#microsoftcacheredisenterprise).

Example:

```yaml
name: azure_cache_redis_enterprise_percentage_cpu
description: "Average percentage cpu usage on an Azure Cache for Redis Enterprise"
resourceType: RedisEnterpriseCache
azureMetricConfiguration:
metricName: usedmemorypercentage
aggregation:
type: Average
resources: # Optional, required when no resource discovery is configured
- cacheName: promitor-redis-enterprise-cache-1
- cacheName: promitor-redis-enterprise-cache-2
resourceDiscoveryGroups: # Optional, requires Promitor Resource Discovery agent (https://promitor.io/concepts/how-it-works#using-resource-discovery)
- name: redis-enterprise-cache-landscape
```
<!-- markdownlint-disable MD033 -->
[&larr; back to metrics declarations](/configuration/v2.x/metrics)<br />
[&larr; back to introduction](/)
<!-- markdownlint-enable -->
39 changes: 39 additions & 0 deletions docs/configuration/v2.x/metrics/virtual-network.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
layout: default
title: Azure Virtual Network Declaration
---

## Azure Virtual Network

![Availability Badge](https://img.shields.io/badge/Available%20Starting-v2.6-green.svg)![Resource Discovery Support Badge](https://img.shields.io/badge/Support%20for%20Resource%20Discovery-Yes-green.svg)

You can declare to scrape an Azure Virtual Network via the `VirtualNetwork` resource
type.

When using declared resources, the following fields need to be provided:

- `virtualNetworkName` - The name of the Azure Virtual Network resource

All supported metrics are documented in the official [Azure Monitor documentation](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported#microsoftnetworkvirtualnetworks).

Example:

```yaml
name: azure_virtual_network_ddos_attack
description: "Indication whether or not there is a DDOS attack on the Azure Virtual Network"
resourceType: VirtualNetwork
azureMetricConfiguration:
metricName: IfUnderDDoSAttack
aggregation:
type: Maximum
resources: # Optional, required when no resource discovery is configured
- virtualNetworkName: promitor-virtual-network-1
- virtualNetworkName: promitor-virtual-network-2
resourceDiscoveryGroups: # Optional, requires Promitor Resource Discovery agent (https://promitor.io/concepts/how-it-works#using-resource-discovery)
- name: virtual-network-landscape
```
<!-- markdownlint-disable MD033 -->
[&larr; back to metrics declarations](/configuration/v2.x/metrics)<br />
[&larr; back to introduction](/)
<!-- markdownlint-enable -->
5 changes: 5 additions & 0 deletions docs/configuration/v2.x/resource-discovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,12 +93,15 @@ Dynamic resource discovery is supported for the following scrapers:
- [Azure App Plan](metrics/app-plan)
- [Azure Automation](metrics/automation-account)
- [Azure Cache for Redis](metrics/redis-cache)
- [Azure Cache for Redis Enterprise](metrics/redis-enterprise-cache)
- [Azure Content Delivery Network (CDN)](metrics/cdn)
- [Azure Container Instances](metrics/container-instances)
- [Azure Container Registry](metrics/container-registry)
- [Azure Cosmos DB](metrics/cosmos-db)
- [Azure Data Factory](metrics/data-factory)
- [Azure Data Share](metrics/data-share)
- [Azure Database for PostgreSQL](metrics/postgresql)
- [Azure Database for MariaDB](metrics/maria-db)
- [Azure Event Hubs](metrics/event-hubs)
- [Azure Express Route Circuit](metrics/express-route-circuit)
- [Azure Front Door](metrics/front-door)
Expand All @@ -107,6 +110,7 @@ Dynamic resource discovery is supported for the following scrapers:
- [Azure IoT Hub Device Provisioning Service (DPS)](metrics/iot-hub-device-provisioning-service)
- [Azure Key Vault](metrics/key-vault)
- [Azure Kubernetes Service](metrics/kubernetes)
- [Azure Load Balancer](metrics/load-balancer)
- [Azure Logic Apps](metrics/logic-apps)
- [Azure Monitor Autoscale](metrics/monitor-autoscale)
- [Azure Network Gateway](metrics/network-gateway)
Expand All @@ -121,6 +125,7 @@ Dynamic resource discovery is supported for the following scrapers:
- [Azure Synapse (Workspace)](metrics/synapse-workspace)
- [Azure Virtual Machine](metrics/virtual-machine)
- [Azure Virtual Machine Scale Set (VMSS)](metrics/virtual-machine-scale-set)
- [Azure Virtual Network](metrics/virtual-network)
- [Azure Web App](metrics/web-app)

[&larr; back](/)
Loading

0 comments on commit 9ee0930

Please sign in to comment.