v1.2.1

Release 1.2.1 - Validator Performance History Hotfix

• Fixed a bug wherein validators overwrote their miner performance history cache. This occurred when the base neuron init was called, which for some reason called save_state before calling load_state. Added an explicit load of the MinerPerformanceManager object prior to this.
• Also updating the cached MinerPerformanceManager once per challenge to ensure no data are lost if a validator crashes or restarts.

v1.2.0

Release 1.2.0 - Incentive Mechanism Enhancement

TLDR

Reward = AUC**2
Score Update = EMA with α = 0.01.

Rewards

Miners rewards are computed as the squared ROC AUC of (up to) their last 100 predictions.

$$ r_i = \text{AUC}_i^2 $$

$$ \text{AUC} = \int_{0}^{1} \text{TPR}(t) \cdot \frac{d}{dt}\text{FPR}(t) dt $$

where t is the classification threshold and

$$\text{TPR} = \frac{\text{TP}}{\text{TP} + \text{FN}}$$

and

$$\text{ FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}}$$

ROC AUC, or Receiver Operating Characteristic Area Under the Curve, is a powerful metric for evaluating binary classification models. It was chosen for its following strenghts:

• Robustness to class imbalance: performs well even when one class is more common than the other (to avoid volatilitiy in cases where miners receieve a series of all fake or all real images).
• Interpretability: provides a single scalar value between 0 and 1, where 1 indicates perfect classification.
• Sensitivity and specificity balance: considers both true positive and false positive rates.

These characteristics make ROC AUC an ideal metric for evaluating miner performance in our subnet, where accurate classification of real and synthetic images is crucial.

Scores

Validators set weights based on a score vector they use to keep track of miner performance.

For each challenge t, a validator will randomly sample 50 miners, send them an image, and compute their rewards C based on their responses. These reward values are then used to update the validator's score vector V using an exponential moving average (EMA) with α = 0.01.

$$ V_t = 0.01 \cdot C_t + 0.99 \cdot V_{t-1} $$

A low α value places emphasis on a miner's historical performance, adding additional smoothing to avoid having a single prediction cause significant score fluctuations.

v.1.1.1

Release 1.1.1 - Deepfake Detectors

TL;DR

• No immediate actions required
• Miners, you now have cleaner model training and miner deployment infrastructure
• Validators, please update your hardware per our updated min-compute.yaml at your convenience in preparation for our next release. We will be reaching out to you all individually with specific instructions regarding this release shortly, thank you in advance!

Validator Updates

Minimum compute requirements have been updated preemptively in light of our upcoming 1.2.0 release. More on this soon!

Miner Updates (QOL)

This release introduces significant revamping of the base_miner/ directory, notably standardizing how detectors are instantiated and invoked. Note that in the following, we use the term detector to mean any system orchestrating one or more models to perform binary deepfake classification.
These changes make it easier to configure existing detectors to deploy as miners (e.g. quickly swap out pretrained model weights), or extend our base miners with new contributions.

Key changes -- Detector and Miner Modularization

• Single consolidated miner.py, with dynamic, configurable detector initialization using new arguments in miner.env to select the DeepfakeDetector(ABC) subclass and YAML configs to load
• New deepfake_detectors/ with detector implementations that perform model/config loading and inference invocation, subclassing a common DeepfakeDetector(ABC) abstract base class.
  • NPRDetector(DeepfakeDetector) , UCFDetector(DeepfakeDetector) , CAMODetector(DeepfakeDetector) implementations
  • Starter configs for initializing detector classes in base_miner/deepfake_detectors/configs/
• New gating_mechanisms/ with gate implementations that handle content-aware image preprocessing, subclassing a common Gate(ABC) abstract base class.
  • Moved topic specific preprocessing from detectors (e.g. face utils) to gates
  • GatingMechanism class which contains a configurable number of Gates, each of which is invoked in a standard fashion.
• New registry.py which uses a factory method design pattern to instantiate detector and gate subclasses with Registry
• UCF Training Improvements:
  • Cleaner outputs, now saved to single files and in a single folder per run.
  • Training run config now saved before first epoch, for reproducibility and reconstructing correct layer shapes for inference.
    • These configs are loaded in UCFDetector(DeepfakeDetector)
  • New cli args for QOL and hyperparameter tuning (epochs, num dataloader workers)
  • New error messages for diagnosing UCF parameter issues affecting output layer shapes.
  • Renamed training script to train_detector.py to be consistent with NPR training
  • Simplified Dataset loading and splitting

Other

• Updated bittensor version 7.3.1 -> 7.4.0
• Relocated training dataset generation pipelines to the bitmind-utils repository
• Deleted unused UCF legacy scripts.
• Updated docs/readmes to reflect changes in this release

v1.1.0

Release 1.1.0 - UCF and CAMO Base Miners

TL;DR

Miner Upgrade: We recommend switching to the improved camo_miner.py neuron for enhanced deepfake detection performance and competitive advantage.
Hardware Requirements: Minimum GTX 1070 GPU or rough equivalent required. See min-compute.yml for details.
New Datasets: Access to updated datasets for training.
Optimized Setup Scripts: Faster, tailored scripts for miners and validators now available.

We've also updated our runpod and tensordock quickstart tutorials to reflect the new base models:
Tensordock Quickstart
RunPod Quickstart

⛏️ New Base Miner

Our base miner upgrade comes in two parts:

1. New foundation model using the UCF architecture (Uncovering Common Features), the top scoring deepfake detection model in the “Top 3” metric of DeepfakeBench

• UCF Paper (ICCV 2023)
• DeepfakeBench Repo
• Pretrained model weights available on the BitMind HuggingFace

2. We’re also releasing CAMO (Content Aware Model Orchestration), a generalized framework for creating “hard mixture of expert” models for deepfake detection

• Our initial implementation of CAMO is our most performant base miner yet, featuring an expert deepfake detector (for face image content) and a generalist deepfake detector (for all content). Both components are finetuned UCF models.
• We’ll be extending this architecture in a number of ways - to read about our future work on CAMO you can check out our blog, or join us in our Discord server on Monday at 10am PST for Miner Mondays
• Preprocessed training datasets and corresponding pipeline code are available in our subnet repo and HuggingFace
  • Face expert training datasets (Cropped, aligned, and transformed faces only)

🚀 QOL Upgrades

New Environment Setup Flow:

1. conda create -y -n bitmind python=3.10 && conda activate bitmind
2. ./setup_validator_env.sh or ./setup_miner_env.sh

❗ Note that when you run the new setup scripts, they will overwrite your existing .env file in order to add a couple new fields.

Reason for this change:
With the addition of UCF and CAMO comes additional python dependencies for miners, some of which take a while to install.
To avoid burdening validators with extra installs, we’ve created separate scripts that only install the necessary packages for the corresponding neuron type.

v1.0.2

Release 1.0.2 - QOL Improvements

QOL Upgrades

• Auto-update for both validators and miners.
• Self-healing for both validators and miners, restarts neuron every 6 hours to help keep processes in a healthy state
• Reduced validator log clutter
  • Before: Lots of innocuous warnings and ugly progress bars from pytorch, huggingface, etc. that made logs difficult to parse
  • Now: Silenced warnings and removed progress bars, keeping only information essential to understanding validator behavior/health

Validator Technical Improvements:

Improved validator image generation pipeline

Before

• The validator prompt generation pipline produced annotations with inconsistent white spacing and punctuation between prompt responses.
• Our image generation pipline attempted to generate images from annotations that exceeded the maximum allowed length of 77 tokens for CLIPTokenizer, the tokenizer for the Stable Diffusion XL, Mobius, and RealVisXL_V4.0 diffusion models used by validators. These long annotations previously triggered warnings.
• Previously in the validator forward function, resampling was not performed if a NAN value was present in any field of a real image sample dict (e.g. Image id).

Now

• Improved BLIP-2 outputs to be cleaner with spaces between each prompt and periods at the end of responses.
• Synthetic image generation now first truncates all descriptive information into the maximum tokenizer token limit upfront, to ensure compliance with HuggingFace diffusion model tokenizers and avoid warnings.
• Validator forward function now checks for NAN image metadata after each sampling, annotation, and image generation step. A retry with a resampled real image is performed if data is invalid.