Feature Request: Enhanced Submission Analytics & Insights Dashboard

[aditisingh02]

Summary

The current analytics module in EvalAI provides only basic submission counts and participant metrics. This feature request proposes expanding the analytics capabilities to provide challenge hosts with deeper insights into submission patterns, evaluation performance, and participant engagement.

Problem Statement

Currently, the analytics module (apps/analytics/views.py) offers limited functionality:

• Basic participant and team counts
• Simple submission counts (daily, weekly, monthly)
• Last submission timestamps

Challenge hosts lack visibility into:

• Submission success/failure trends
• Evaluation performance bottlenecks
• Participant engagement patterns over time
• Leaderboard dynamics and progression

Proposed Features

1. Submission Success/Failure Rates per Phase

Description: Track and visualize the ratio of successful vs. failed submissions for each challenge phase.

Metrics to include:

• Success rate percentage per phase
• Failure breakdown by error type (timeout, runtime error, invalid format, etc.)
• Trend analysis over time (daily/weekly)

API Endpoint Example:

GET /api/analytics/challenges/{challenge_pk}/phases/{phase_pk}/submission-rates/

Response Example:

{
  "phase_id": 1,
  "phase_name": "Test Phase",
  "total_submissions": 1500,
  "successful": 1200,
  "failed": 300,
  "success_rate": 80.0,
  "failure_breakdown": {
    "timeout": 120,
    "runtime_error": 100,
    "invalid_format": 50,
    "other": 30
  }
}

---

2. Average Evaluation Time Tracking

Description: Monitor and display average evaluation times to help hosts identify performance bottlenecks.

Metrics to include:

• Average evaluation time per phase
• 50th, 90th, 99th percentile evaluation times
• Evaluation time trends over the past 7/30 days
• Comparison across different phases

API Endpoint Example:

GET /api/analytics/challenges/{challenge_pk}/evaluation-times/

Response Example:

{
  "challenge_id": 1,
  "phases": [
    {
      "phase_id": 1,
      "phase_name": "Dev Phase",
      "avg_evaluation_time_seconds": 45.2,
      "p50": 30.0,
      "p90": 120.0,
      "p99": 300.0
    }
  ],
  "trend": [
    {"date": "2026-01-10", "avg_time": 42.5},
    {"date": "2026-01-11", "avg_time": 48.3}
  ]
}

---

3. Participant Activity Heatmaps

Description: Visualize when participants are most active to help hosts optimize challenge scheduling and support availability.

Metrics to include:

• Submissions by hour of day (UTC)
• Submissions by day of week
• Geographic distribution of activity (if available)
• Peak activity periods

API Endpoint Example:

GET /api/analytics/challenges/{challenge_pk}/activity-heatmap/

Response Example:

{
  "challenge_id": 1,
  "heatmap": {
    "Monday": {"00": 5, "01": 3, "02": 2, ..., "23": 12},
    "Tuesday": {"00": 4, "01": 2, ..., "23": 15}
  },
  "peak_hours": ["14:00-16:00 UTC", "20:00-22:00 UTC"],
  "most_active_day": "Saturday"
}

---

4. Leaderboard Progression Charts

Description: Track how the leaderboard evolves over time, showing score improvements and ranking changes.

Metrics to include:

• Top-N teams' score progression over time
• Rank changes history for specific teams
• Best score improvements per time period
• New entries to top-N tracking

API Endpoint Example:

GET /api/analytics/challenges/{challenge_pk}/phases/{phase_pk}/leaderboard-progression/

Response Example:

{
  "phase_id": 1,
  "top_teams_progression": [
    {
      "team_name": "Team Alpha",
      "progression": [
        {"date": "2026-01-01", "score": 0.85, "rank": 3},
        {"date": "2026-01-05", "score": 0.89, "rank": 1}
      ]
    }
  ],
  "score_improvements": {
    "biggest_jump": {"team": "Team Beta", "improvement": 0.15},
    "avg_improvement": 0.03
  }
}

---

Technical Implementation

Backend Changes

1. New Models (optional):

   • SubmissionAnalytics - Store aggregated analytics data
   • EvaluationMetrics - Track evaluation performance

2. New Serializers in apps/analytics/serializers.py:

   • SubmissionRateSerializer
   • EvaluationTimeSerializer
   • ActivityHeatmapSerializer
   • LeaderboardProgressionSerializer

3. New Views in apps/analytics/views.py:

   • get_submission_success_rates()
   • get_evaluation_time_metrics()
   • get_participant_activity_heatmap()
   • get_leaderboard_progression()

4. Database Queries:

   • Leverage Django ORM aggregation functions
   • Consider caching for frequently accessed analytics
   • Use background tasks (Celery) for heavy computations

Frontend Changes

1. New Components in frontend_v2/src/app/components/:

   • analytics-dashboard/
   • submission-rate-chart/
   • evaluation-time-chart/
   • activity-heatmap/
   • leaderboard-progression/

2. Visualization Libraries:

   • Use Chart.js or D3.js for interactive charts
   • Heatmap.js for activity heatmaps

---

Mockups

Submission Success/Failure Rates

┌─────────────────────────────────────────┐
│  Dev Phase - Submission Results         │
│  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━    │
│  ████████████████████░░░░░ 80% Success  │
│                                         │
│  Failed: 300 (20%)                      │
│  ├── Timeout: 120                       │
│  ├── Runtime Error: 100                 │
│  └── Invalid Format: 80                 │
└─────────────────────────────────────────┘

Activity Heatmap

        Mon Tue Wed Thu Fri Sat Sun
  00:00  ░   ░   ░   ░   ░   █   █
  06:00  ░   ░   ░   ░   ░   ▓   ▓
  12:00  ▓   ▓   ▓   ▓   ▓   █   █
  18:00  █   █   █   █   █   ▓   ▓
  
  ░ Low  ▓ Medium  █ High

---

Benefits

• For Challenge Hosts:

  • Better understanding of participant behavior
  • Identify evaluation bottlenecks
  • Optimize challenge phase timings
  • Data-driven decision making

• For Participants:

  • Understand their progress relative to the leaderboard
  • Know optimal times to submit

• For Platform:

  • Differentiate from competitors
  • Increase host engagement and retention

[aditisingh02]

I'm interested in implementing this feature. I'll start with the backend API endpoints and then move to the frontend dashboard. Please assign this to me.

[RitikaSen27]

Hi! I’d like to work on this issue and submit a PR.
Please let me know if that’s okay.

[OdysseasKalaitsidis]

Hi @aditisingh02, I’m interested in contributing to the frontend dashboard for this issue. Can we coordinate so we don’t overlap?

[RitikaSen27]

Hi! I’ve explored the analytics module (apps/analytics/views.py).

As a first step, I’d like to add a small UI-focused improvement:
a submission trend summary showing

• total submissions
• submissions in the last 7 days
• submissions in the last 30 days

This would give challenge hosts a quick view of recent engagement
without changing existing analytics logic.

Please let me know if this scope works, or if you’d prefer a different
metric to start with. Thanks!