Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 70 additions & 58 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,89 +1,101 @@
# kube-job-notifier
For kubernetes job notification tool

## Description
- Notification Kubernetes Job start, success, and failure.
A tool for monitoring Kubernetes job execution status and sending notifications to Slack and Datadog.

## Usage
## Features

### Notification setting(Slack only)
- Please set environment variable
- Notifications for Kubernetes job start, success, and failure
- Slack notifications with log attachments
- Datadog service check notifications
- Support for multiple container log collection
- Per-job notification customization
- Easy deployment with Helm charts

## Installation

### Installation with Helm (Recommended)

```bash
helm repo add kube-job-notifier https://yutachaos.github.io/kube-job-notifier/
helm install kube-job-notifier/kube-job-notifier --generate-name
```

### Installation with Manifests

1. Apply the manifests:
```bash
kubectl apply -f manifests/sample/
```

2. Verify the deployment:
```bash
kubectl get po
```

## Configuration

### Slack Notification Settings

Configure using environment variables:

```bash
export SLACK_TOKEN=YOUR_SLACK_TOKEN
export SLACK_CHANNEL=YOUR_NOTIFICATION_CHANNEL_ID
export SLACK_STARTED_NOTIFY=true # OPTIONAL DEFAULT true
export SLACK_SUCCEEDED_NOTIFY=true # OPTIONAL DEFAULT true
export SLACK_FAILED_NOTIFY=true # OPTIONAL DEFAULT true
export SLACK_USERNAME=YOUR_NOTIFICATION_USERNAME # OPTIONAL
export SLACK_SUCCEED_CHANNEL=YOUR_NOTIFICATION_CHANNEL_ID # OPTIONAL
export SLACK_FAILED_CHANNEL=YOUR_NOTIFICATION_CHANNEL_ID # OPTIONAL
export DATADOG_ENABLED=true # OPTIONAL DEFAULT false
export NAMESPACE=KUBERNETES_NAMESPACE # OPTIONAL
export SLACK_STARTED_NOTIFY=true # Optional, default: true
export SLACK_SUCCEEDED_NOTIFY=true # Optional, default: true
export SLACK_FAILED_NOTIFY=true # Optional, default: true
export SLACK_USERNAME=YOUR_NOTIFICATION_USERNAME # Optional
export SLACK_SUCCEED_CHANNEL=YOUR_NOTIFICATION_CHANNEL_ID # Optional
export SLACK_FAILED_CHANNEL=YOUR_NOTIFICATION_CHANNEL_ID # Optional
export DATADOG_ENABLED=true # Optional, default: false
export NAMESPACE=KUBERNETES_NAMESPACE # Optional
```

It will take SLACK_CHANNEL as default channel which may be overwritten by SLACK_SUCCEED_CHANNEL, SLACK_FAILED_CHANNEL environment variables.
### Job Annotation Configuration

Another way of overriding behaviour is using job annotations in k8s. Available job annotations to override are:
You can customize notification channels per job:

```
- kube-job-notifier/default-channel - will be used as channel for a notification if similar success channel annotation is empty
- kube-job-notifier/success-channel - will be used as channel for a success job notification
- kube-job-notifier/started-channel - will be used as channel for a started job notification
- kube-job-notifier/failed-channel - will be used as channel for a failed job notification
kube-job-notifier/default-channel - Default notification channel
kube-job-notifier/success-channel - Channel for successful job notifications
kube-job-notifier/started-channel - Channel for job start notifications
kube-job-notifier/failed-channel - Channel for failed job notifications
```

Also it's possible to suppress notification per job:
You can also suppress notifications:

```
- kube-job-notifier/suppress-success-notification - suppress notification for succesfully finished job even if SLACK_SUCCEEDED_NOTIFY environment variable set to true
- kube-job-notifier/suppress-started-notification - suppress notification when job is started even if SLACK_STARTED_NOTIFY environment variable set to true
- kube-job-notifier/suppress-failed-notification - suppress notification when job is failed even if SLACK_FAILED_NOTIFY environment variable set to true
```
#### slack permissions
- Required permission above.
```
chat:write
files:write
kube-job-notifier/suppress-success-notification - Suppress success notifications
kube-job-notifier/suppress-started-notification - Suppress start notifications
kube-job-notifier/suppress-failed-notification - Suppress failure notifications
```

### Event subscription setting(Current Datadog support only)
- Datadog service checks are sent when the Job succeeds or fails.
- More information https://docs.datadoghq.com/developers/service_checks/dogstatsd_service_checks_submission/
### Slack Permission Requirements

### Job with multiple containers logging
The following permissions are required:
- `chat:write`
- `files:write`

By default for cron jobs logs are attached from container with the same name as a cron job. This can be overwritten by adding *kube-job-notifier/log-mode* annotation.
### Datadog Integration

- *ownerContainer* - get logs only from container with the same name as cron job (default behaviour if annotation is not presented);
- *podOnly* - get logs from the pod, works perfectly with pod with single container;
- *podContainers* - get logs from all pod containers and concatenate them.
- Sends Datadog service checks when jobs succeed or fail
- For more details: https://docs.datadoghq.com/developers/service_checks/dogstatsd_service_checks_submission/

### Run
### Multiple Container Log Collection

#### Local
You can specify log collection mode using the `kube-job-notifier/log-mode` annotation:

`go run *.go -kubeconfig {YOUR_KUBECONFIG_PATH}`
#### Kubernetes
- `ownerContainer` - Get logs only from container with the same name as the job (default)
- `podOnly` - Get logs from the pod, works perfectly with single container pods
- `podContainers` - Get logs from all pod containers and concatenate them

- Run your kubernetes cluster.(Note default namespace is `default`). If change apply namespace, please edit manifest.
- Setting SLACK_TOKEN and SLACK_CHANNEL in manifest/sample/deployment.yaml.
- Apply manifest
`kubectl apply -f manifests/sample/`
- Check running
## Running Locally

```bash
go run *.go -kubeconfig {YOUR_KUBECONFIG_PATH}
```
kubectl get po
NAME READY STATUS RESTARTS AGE
kube-job-notifier-deployment-698fbc8b54-ffk2q 1/1 Running 0 8m12s
```

#### Install with Helm
`helm repo add kube-job-notifier https://yutachaos.github.io/kube-job-notifier/`
`helm install kube-job-notifier/kube-job-notifier --generate-name`

#### Document
## Documentation

https://yutachaos.github.io/kube-job-notifier/
For detailed documentation, visit: https://yutachaos.github.io/kube-job-notifier/

Loading