v1.0.0

This new release brings ~400 new commits with improvements all over the place: docs, features, fixes... QA-Board is used in many projects at Samsung, and we don't have plans to break backward compatibility. Therefore we feel ready to call it this version v1.

If you run into issues with QA-Board, or want to know more about its use-cases, know it's actively developed. Don't hesitate to contact me.

New Features

• Display run badges to e.g. show inference results in QA-Board and link to model training logs
• Compare versus runs from different projects
• Soft-deletes to keep run metadata but remove all/some files
• Parallel sampling for auto-tuning
• User accounts, required to trigger experiments from the app.
• Per-user custom tests in the web application.
• Move runs to an existing or new batch
• Built-in distributed workers with celery, and monitoring with flower.
• Redo: all a batch's run, or just failed ones, or per-run... and with the same parameters as the original.
• Usage outside of git repos: support GIT_MESSAGE / GIT_AUTHORED_DATETIME / GIT_COMMIT / GIT_BRANCH in ENV.
• Support for default database per batches.yaml file
• Use indexing when using matrix configurations, e.g. ${matrix.value[0]}
• Input manifests support, and a --strict mode in bit-accuracy checks that forbids unexpected output files
• distributed workers via Jenkins
• Custom bit-accuracy compare function to e.g. ignore metadata
• More flexible per-input configurations in batches
• Can use globs to define in qaboard.yaml inputs.batches. In this case it can be helpful to enable the optional flag allow_duplicate_batches.
• Internal: for "user-side" DAG/pipelines make it possible to call e.g. CI=true GIT_COMMIT=xxxxx qa batch --action-on-existing=assert-exists

UX improvements

• Tons of improvements to the logs, status indicators and error messages. Both in the CLI and web application.
• We tried to make the defaults of everything (UI, config...) more natural.
• Tuning analysis / tradeoffs tab enabled with runs started on the CLI, not just with experiments from the app
• UI: layout improvements in many places
• Image viewer: No animations to make the diff feel smoother, tooltip with real image values, hide the reference if same as new. Allow ref images with different dimensions, and behave nicely.
• File viewer icons
• Filter list of projects
• Scroll to bottom button in logs
• target_passfail metrics to show in green/red instead of a gradient
• Text viewer: toogle side-by-side / inline
• Indicator when identical files are compared
• Filter milestones in the selection menu. Direct links to milestones in the project sidebar.
• Remove columns without metrics from the table view
• Mark pending runs as finished from the UI
• Highlight commit that have milestones and prevent deleting them
• Avoid recoloring plotly plots if we don't overlay new/reference
• ...and many minor things

Performance

• Support for custom postgresql configuration
• Removed re-renders and overhead in the web application
• Transparent pre-compressed gzip files (.gz) to limit storage usage with custom viewers
• Memoize string edit distances

Fixes

• Many minor issues all over the place...
• Gitlab not needed anymore. But if used, added a proxy to get project avatars, and support triggering manual jobs in CI pipelines with tons of jobs...
• hardcoded ports, URL...
• Don't rely on conventions for the output directory locations
• Better uswgi setup: no startup failures, "cheaper" algo, set queue size...
• Fix imports when not running under a QA-Board directory

Upgrades

• Upgraded dependencies of the web application and backend
• Website updated to Docusaurus v2