otc-cloudeye-exporter

A robust Prometheus exporter for Open Telekom Cloud (OTC) Cloud Eye metrics with enterprise-grade features including multi-project support, dynamic namespace selection, and comprehensive logging.

🌟 Key Features

Multi-Project Architecture: Simultaneously collect metrics from multiple OTC projects with automatic project validation
Dynamic Namespace Selection: Configure namespaces statically or override them dynamically via HTTP query parameters
Enterprise Logging: File-based logging with rotation, compression, and configurable levels
Flexible Deployment: Support for both HTTP and HTTPS with customizable ports and TLS configuration
Service Endpoint Mapping: Configurable API endpoints for different OTC services and regions
Error Resilience: Graceful handling of project validation failures and missing endpoints
Resource Management: Automatic cleanup of clients and connections on shutdown

📋 Prerequisites

Go 1.19+ (for building from source)
Valid OTC credentials (Access Key & Secret Key)
Network access to OTC Cloud Eye APIs

🚀 Quick Start

Installation

git clone https://github.com/abdo-farag/otc-cloudeye-exporter.git
cd otc-cloudeye-exporter
go build -o otc-cloudeye-exporter

Basic Configuration

Create the required configuration files in your working directory:

1. `clouds.yml` - Main Configuration

auth:
  access_key: "your_otc_access_key"
  secret_key: "your_otc_secret_key" 
  region: "eu-de"
  # If no project set it will scrap all project for the selected namespace/s
  projects:
    - name: "eu-de_prod"
    - name: "eu_de_staging"
    - name: "eu_de_dev"
    - name: "eu_de"

global:
  metric_path: "/metrics"
  port: 9098
  https_port: 9099
  enable_https: false
  tls_cert: "/path/to/cert.pem"
  tls_key: "/path/to/key.pem"
  namespaces: "SYS.ECS,SYS.EVS,SYS.RDS,SYS.ELB"
  ignore_ssl_verify: true

2. `endpoints.yml` - Service Endpoints

region: "eu-de"
services:
  SYS.ECS: "https://ecs.{eu-de}.otc.t-systems.com"
  SYS.EVS: "https://evs.{eu-de}.otc.t-systems.com" 
  SYS.RDS: "https://rds.{eu-de}.otc.t-systems.com"
  SYS.ELB: "https://elb.{eu-de}.otc.t-systems.com"
  SYS.VPC: "https://vpc.{eu-de}.otc.t-systems.com"
  SYS.OBS: "https://obs.{eu-de}.otc.t-systems.com"

3. `logs.yml` - Logging Configuration

logging:
  - type: FILE
    level: info
    file:
      enabled: true
      filename: "otc-exporter.log"
      encoder: json
      max_size: 10485760    # 10MB
      max_backups: 5
      max_age: 30           # days
      compress: true
  - type: CONSOLE
    level: info
    console:
      enabled: true
      encoder: console

Modify the env file with your tenant values

cat > .env << 'EOF'
export OS_DOMAIN_ID=your_domain_id_here
export OS_DOMAIN_NAME=your_domain_name_here
export OS_ACCESS_KEY=your_access_key_here
export OS_SECRET_KEY=your_secret_key_here
EOF

source .env

Running the Exporter

./otc-cloudeye-exporter -config ./clouds.yml

📊 Usage

Accessing Metrics

Default namespaces: http://localhost:9098/metrics
Custom namespaces: http://localhost:9098/metrics?ns=SYS.ECS,SYS.RDS
HTTPS (if enabled): https://localhost:9099/metrics

Supported OTC Namespaces

The exporter can collect metrics from any OTC Cloud Eye namespace. Configure the namespaces you need in your clouds.yml or specify them dynamically via query parameters.

Common OTC Namespaces and Services Interconnected with Cloud Eye:

Category	Service	Namespace	Reference
Compute	Elastic Cloud Server	`SYS.ECS`	ECS Metrics
	Elastic Cloud Server Agent based Metrics	`AGT.ECS`	ECS Metrics
	Bare Metal Server	`SERVICE.BMS`	BMS Metrics
	Auto Scaling	`SYS.AS`	AS Metrics
Storage	Elastic Volume Service (attached to an ECS or BMS)	`SYS.EVS`	EVS Metrics
	Object Storage Service	`SYS.OBS`	OBS Metrics
	Scalable File Service	`SYS.SFS`	SFS Metrics
	SFS Turbo	`SYS.EFS`	SFS Turbo Metrics
	Cloud Backup and Recovery	`SYS.CBR`	CBR Metrics
Network	Elastic IP and bandwidth	`SYS.VPC`	VPC Metrics
	Elastic Load Balance	`SYS.ELB`	ELB Metrics
	Direct Connect	`SYS.DCAAS`	Direct Connect Metrics
	NAT Gateway	`SYS.NAT`	NAT Gateway Metrics
	Enterprise Router	`SYS.ER`	Enterprise Router Metrics
	Virtual Private Network	`SYS.VPN`	VPN Metrics
Security	Web Application Firewall	`SYS.WAF`	WAF Metrics
	Cloud Firewall	`SYS.CFW`	CFW Metrics
Application	Distributed Message Service	`SYS.DMS`	DMS Metrics
	Distributed Cache Service	`SYS.DCS`	DCS Metrics
	API Gateway	`SYS.APIC`	API Gateway Metrics
Database	Relational Database Service	`SYS.RDS`	RDS Metrics
	Document Database Service	`SYS.DDS`	DDS Metrics
	GeminiDB	`SYS.NoSQL`	GeminiDB Metrics
	GaussDB(for MySQL)	`SYS.GAUSSDB`	GaussDB MySQL Metrics
	GaussDB(for openGauss)	`SYS.GAUSSDBV5`	GaussDB openGauss Metrics
Data Analysis	Data Warehouse Service	`SYS.DWS`	DWS Metrics
	Cloud Search Service	`SYS.ES`	Elasticsearch Metrics
	Data Ingestion Service (DIS)	`SYS.DAYU`	DIS Metrics

Note: This list is not exhaustive. The exporter supports any namespace available in OTC Cloud Eye. Simply add the desired namespaces to your configuration or endpoint mappings.

🔧 Advanced Configuration

HTTPS Configuration

global:
  enable_https: true
  https_port: 9099
  tls_cert: "/etc/ssl/certs/exporter.crt"
  tls_key: "/etc/ssl/private/exporter.key"

Multi-Region Setup

For multiple regions, run separate exporter instances with different configurations:

# eu-de-config.yml
auth:
  region: "eu-de"
  # ... other config

# eu-nl-config.yml  
auth:
  region: "eu-nl"
  # ... other config

Project Validation

The exporter automatically validates that configured projects exist in the specified region before initializing clients. Invalid projects are logged and skipped, allowing the exporter to continue with valid projects.

🔍 Monitoring & Observability

Logs

The exporter provides detailed logging including:

Project validation results
Client initialization status
Metric collection activities
Error conditions and warnings

Health Check

The exporter serves on the configured port and will respond to health checks on the metrics endpoint.

🐳 Docker Deployment

cat > .env-docker << 'EOF'
OS_DOMAIN_ID=your_domain_id_here
OS_DOMAIN_NAME=your_domain_name_here
OS_ACCESS_KEY=your_access_key_here
OS_SECRET_KEY=your_secret_key_here
EOF

docker run --rm --name cloudeye-exporter -p 9098:9098 -p 9099:9099 \
--env-file .env-docker \
ghcr.io/abdo-farag/otc-cloudeye-exporter:latest

Or

source .env

docker run --rm --name cloudeye-exporter -p 9098:9098 -p 9099:9099 \
  -e OS_DOMAIN_ID=${OS_DOMAIN_ID} \
  -e OS_DOMAIN_NAME=${OS_DOMAIN_NAME} \
  -e OS_ACCESS_KEY=${OS_ACCESS_KEY} \
  -e OS_SECRET_KEY=${OS_SECRET_KEY} \
  ghcr.io/abdo-farag/otc-cloudeye-exporter:latest

using env file

📈 Prometheus Integration

Prometheus Configuration

# prometheus.yml
scrape_configs:
  - job_name: 'otc-cloudeye'
    static_configs:
      - targets: ['localhost:9098']
    scrape_interval: 60s
    metrics_path: '/metrics'
    params:
      ns: ['SYS.ECS,SYS.EVS,SYS.RDS']

Grafana Alloy Configuration

prometheus.remote_write "mimir" {
  endpoint {
    url = "http://mimir:9009/api/v1/push"
  }
}

prometheus.scrape "cloudeye_exporter" {
  targets = [
    { __address__ = "<otc_cloudeye_exporter>:9098" },
  ]

  metrics_path    = "/metrics"
  scrape_interval = "10s"
  job_name        = "otc_cloudeye_exporter"

  params = {
    ns = ["SYS.ECS,SYS.VPC,SYS.ELB"],
  }

  forward_to = [prometheus.remote_write.mimir.receiver]
}

Grafana Dashboard

Import metrics using the standard Prometheus data source. Key metric patterns:

otc_cloudeye_* - All OTC Cloud Eye metrics
Labels include: project, namespace, resource_id, region

🛠️ Troubleshooting

Common Issues

No metrics appearing:

Check log files for authentication errors
Verify project names exist in the configured region
Ensure network connectivity to OTC APIs

Project validation failures:

Confirm project names are exact matches (case-sensitive)
Verify credentials have permissions for the projects
Check region configuration matches project locations

SSL/TLS issues:

Set ignore_ssl_verify: true for development environments
Ensure proper certificate configuration for production HTTPS

Debug Mode

Enable debug logging in logs.yml:

logging:
  - type: CONSOLE
    level: debug
    console:
      enabled: true
      encoder: "CONSOLE"
      time_format: "02.01.2006 15:04:05"

🤝 Contributing

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

📄 License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

🙏 Acknowledgments

Open Telekom Cloud (OTC) API documentation and community support
Powered by Huawei Cloud SDK for Go v3
Built with Prometheus Go client library

Name		Name	Last commit message	Last commit date
Latest commit History 24 Commits
.github/workflows		.github/workflows
cmd/otc-cloudeye-exporter		cmd/otc-cloudeye-exporter
helm/otc-cloudeye-exporter		helm/otc-cloudeye-exporter
internal		internal
.env.example		.env.example
.gitignore		.gitignore
.goreleaser.yaml		.goreleaser.yaml
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
clouds.yml		clouds.yml
endpoints.yml		endpoints.yml
go.mod		go.mod
go.sum		go.sum
logs.yml		logs.yml

License

abdo-farag/otc-cloudeye-exporter

Folders and files

Latest commit

History

Repository files navigation