This directory contains cluster configuration profiles optimized for development and testing environments.
- Profile:
small-cluster.yaml - Use Case: Local development, testing, small datasets
- Resources: Minimal compute and memory allocation
- Cost: Low cost for development work
- Profile:
medium-cluster.yaml - Use Case: Integration testing, medium datasets
- Resources: Moderate compute and memory allocation
- Cost: Balanced cost for testing workflows
# Create a small development cluster
dataproc-mcp create_cluster_from_profile \
--profile development/small-cluster \
--cluster-name my-dev-cluster
# Create a medium development cluster
dataproc-mcp create_cluster_from_profile \
--profile development/medium-cluster \
--cluster-name my-test-cluster- Auto-scaling: Enabled for cost optimization
- Preemptible instances: Used to reduce costs
- Development tools: Pre-installed debugging and monitoring tools
- Quick startup: Optimized for fast cluster creation
- Auto-deletion: Configured for automatic cleanup
- Use preemptible instances for cost savings
- Enable auto-scaling to handle variable workloads
- Set auto-deletion timers to prevent forgotten clusters
- Use smaller machine types for development work
- Monitor costs regularly during development
For production workloads, see the production profiles.