Skip to content

Float32 to uint8 conversion in batch processing causes quantization errors #152

@Dipet

Description

@Dipet

Problem

Several transforms (e.g., MedianBlur, and potentially others like GaussianBlur, Blur, etc.) use the @uint8_io decorator or explicit from_float/to_float conversions to work with cv2 functions that require uint8 input. When the user passes float32 images, the pipeline converts float32 → uint8 → process → uint8 → float32, introducing quantization error (up to ±1/255 per round-trip).

For batch processing (apply_to_images), this conversion is done once for the whole batch, but the fundamental precision loss remains.

Proposal

Add a flag (e.g., at the transform level or Compose level) that lets users choose between:

  1. Fast path (default, current behavior): Convert float32 to uint8 before processing. Fastest, but introduces quantization artifacts.
  2. Quality path: Process in float32 directly where the underlying cv2 function supports it (e.g., cv2.medianBlur supports CV_32F for ksize ≤ 5). Slower, but preserves full precision.

Scope

This affects any transform that uses @uint8_io or manual float32→uint8 conversion. A general solution would benefit multiple transforms beyond just MedianBlur, including:

  • Blur / GaussianBlur / AdvancedBlur
  • GlassBlur / Defocus / ZoomBlur
  • Any other pixel-level transforms that force uint8 conversion

Notes

  • Some cv2 functions only support uint8 for certain parameter ranges (e.g., cv2.medianBlur only supports float32 for ksize ≤ 5). The quality path would need to handle these constraints gracefully.
  • Need to decide on the API: per-transform flag, global Compose-level setting, or both.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions