Move to a better update strategy for Elastic search

[SubhadityaMukherjee]

We want to eventually move away from the REST API having to trigger the elastic search update. For this we intend to consider tools like logstash. For example, if something is updated, we want to use logstash's monitoring to automatically trigger an ES update.

(We do not need to use logstash, but this seems like a decent option for now)

Since this is a bit complicated, I will try to do this in steps.

1. Figure out ES indexes we maintain (and their fields) and what they are used for.
2. Set up monitoring using logstash for the indexes
3. Look at the current update strategy and use the monitoring from step 2 to automatically trigger an ES update

[SubhadityaMukherjee]

Step 1 is figuring out the ES indexes that we maintain and their fields. These are present in https://github.com/openml/OpenML/tree/develop/openml_OS/models/api/v1/*.php

Full disclaimer , I'm not too good with php and I used an Ai agent to parse these files and figure out the indexes.

1. data Index

Purpose: Stores dataset metadata, quality metrics, and feature information
Primary Key: data_id (dataset ID)

Key Fields:

• data_id (long) - Dataset identifier
• name (text) - Dataset name with snowball analyzer
• exact_name (keyword) - Exact dataset name for filtering
• version (float) - Dataset version number
• description (text) - Dataset description with snowball analyzer
• format (keyword) - Data format (e.g., ARFF)
• uploader (text) - Uploader name
• uploader_id (long) - Uploader user ID
• date (date) - Upload date
• last_update (date) - Last modification date
• visibility (keyword) - Public/private visibility
• status (keyword) - Dataset status (active, in_preparation, etc.)
• runs (long) - Number of runs using this dataset
• qualities (object) - Dataset quality metrics
• features (nested) - Feature information including types, statistics
• tags (nested) - Associated tags with uploader information
• suggest (completion) - Autocomplete suggestions

Usage: Dataset search, quality analysis, feature exploration, dataset discovery

2. flow Index

Purpose: Stores machine learning algorithm and flow definitions
Primary Key: flow_id (implementation ID)

Key Fields:

• flow_id (long) - Flow identifier
• name (text) - Flow name with snowball analyzer
• exact_name (keyword) - Exact flow name
• version (float) - Flow version
• external_version - External version identifier
• description (text) - Flow description
• full_description (text) - Detailed flow description
• uploader (text) - Uploader name
• uploader_id (long) - Uploader user ID
• date (date) - Upload date
• runs (long) - Number of runs using this flow
• qualities (object) - Flow quality metrics
• tags (nested) - Associated tags
• components (nested) - Flow component information
• parameters (nested) - Flow parameter definitions
• suggest (completion) - Autocomplete suggestions

Usage: Flow search, algorithm discovery, parameter exploration

3. task Index

Purpose: Stores machine learning task definitions and configurations
Primary Key: task_id (task ID)

Key Fields:

• task_id (long) - Task identifier
• runs (long) - Number of runs for this task
• tasktype (object) - Task type information (tt_id, name)
• date (date) - Creation date
• uploader_id (long) - Creator user ID
• visibility (keyword) - Public/private visibility
• embargo_end_date (date) - Embargo expiration date
• tags (nested) - Associated tags
• suggest (completion) - Autocomplete suggestions
• nr_of_issues (long) - Number of reported issues
• nr_of_downvotes (long) - Number of downvotes
• nr_of_likes (long) - Number of likes
• nr_of_downloads (long) - Number of downloads
• reach (long) - Impact/reach metric
• reuse (long) - Reuse metric
• impact (long) - Overall impact score

Usage: Task discovery, task type filtering, task evaluation

4. run Index

Purpose: Stores execution results, evaluations, and output files
Primary Key: run_id (run ID)

Key Fields:

• run_id (long) - Run identifier
• uploader (text) - Uploader name
• uploader_id (long) - Uploader user ID
• run_task (object) - Associated task information
• run_flow (object) - Associated flow information with parameters
• date (date) - Execution date
• output_files (object) - Output file URLs and formats
• evaluations (nested) - Evaluation results with measures and values
• error (text) - Error information
• error_message (text) - Detailed error message
• visibility (keyword) - Public/private visibility
• tags (nested) - Associated tags

Usage: Run search, performance comparison, result analysis

5. user Index

Purpose: Stores user profiles and activity metrics
Primary Key: user_id (user ID)

Key Fields:

• user_id (long) - User identifier
• first_name (text) - User's first name
• last_name (text) - User's last name
• email (text) - User email
• company (text) - User's company
• country (text) - User's country
• bio (text) - User biography
• image - Profile image
• date (date) - Account creation date
• visibility (keyword) - Profile visibility
• gamification_visibility - Gamification settings
• nr_of_uploads (long) - Total uploads
• datasets_uploaded (long) - Number of datasets uploaded
• flows_uploaded (long) - Number of flows uploaded
• tasks_uploaded (long) - Number of tasks uploaded
• runs_uploaded (long) - Number of runs uploaded
• runs_on_datasets (long) - Runs on other users' datasets
• runs_on_flows (long) - Runs using other users' flows
• nr_of_likes (long) - Total likes received
• nr_of_downloads (long) - Total downloads received
• activity (long) - User activity score
• reach (long) - User reach/impact
• badges (array) - User badges
• suggest (completion) - Autocomplete suggestions

Usage: User search, activity tracking, leaderboard functionality

6. study Index

Purpose: Stores collections of related datasets, tasks, flows, and runs
Primary Key: study_id (study ID)

Key Fields:

• study_id (long) - Study identifier
• name (text) - Study name
• alias - Study alias
• description (text) - Study description
• date (date) - Creation date
• uploader_id (long) - Creator user ID
• uploader (text) - Creator name
• visibility (keyword) - Public/private visibility
• study_type (text) - Type of study
• legacy (text) - Legacy flag
• datasets_included (long) - Number of datasets
• tasks_included (long) - Number of tasks
• flows_included (long) - Number of flows
• runs_included (long) - Number of runs
• suggest (completion) - Autocomplete suggestions

Usage: Study discovery, collection management, related resource grouping

7. task_type Index

Purpose: Stores task type definitions and input/output specifications
Primary Key: tt_id (task type ID)

Key Fields:

• tt_id (long) - Task type identifier
• name (text) - Task type name
• description (text) - Task type description
• tasks (long) - Number of tasks of this type
• date (date) - Creation date
• input (array) - Input specifications
• suggest (completion) - Autocomplete suggestions

Usage: Task type filtering, input/output specification lookup

8. measure Index

Purpose: Stores evaluation measures, procedures, and quality metrics
Primary Key: Various (procedure_id, eval_id, quality_id)

Key Fields:

• proc_id (long) - Procedure identifier
• eval_id (keyword) - Evaluation measure identifier
• quality_id (keyword) - Quality metric identifier
• measure_type (text) - Type (estimation_procedure, evaluation_measure, data_quality, flow_quality)
• name (text) - Measure name
• description (text) - Measure description
• task_type (long) - Associated task type
• folds (long) - Number of folds (for procedures)
• repeats (long) - Number of repeats (for procedures)
• min/max (float) - Value bounds (for measures)
• unit (text) - Measurement unit
• higherIsBetter (boolean) - Whether higher values are better
• suggest (completion) - Autocomplete suggestions

Usage: Measure selection, evaluation criteria lookup

9. download Index

Purpose: Tracks downloads of knowledge pieces
Primary Key: download_id (download record ID)

Key Fields:

• download_id (long) - Download identifier
• user_id (long) - Downloading user
• knowledge_type (text) - Type of knowledge piece
• knowledge_id (long) - ID of downloaded item
• count (long) - Download count
• time (date) - Download timestamp

Usage: Download analytics, popularity tracking

10. like Index

Purpose: Tracks likes/votes for knowledge pieces
Primary Key: like_id (like record ID)

Key Fields:

• like_id (long) - Like identifier
• user_id (long) - Liking user
• knowledge_type (text) - Type of knowledge piece
• knowledge_id (long) - ID of liked item
• time (date) - Like timestamp

Usage: Popularity tracking, community feedback

11. downvote Index

Purpose: Tracks downvotes and issues reported for knowledge pieces
Primary Key: downvote_id (downvote record ID)

Key Fields:

• downvote_id (long) - Downvote identifier
• user_id (long) - Reporting user
• knowledge_type (text) - Type of knowledge piece
• knowledge_id (long) - ID of item with issue
• reason (text) - Description of the issue
• reason_id (long) - Reason category ID
• original (text) - Original content
• count (long) - Downvote count
• time (date) - Downvote timestamp

Usage: Quality control, issue tracking, content moderation

[PGijsbers]

The Elastic Search indices are also available under https://es.openml.org (see e.g., https://es.openml.org/data/). You can see that the properties are incomplete. For example, the data index also has impact and nr_of_downloads, among others. I think it would be good to evaluate which of these fields are actually still needed. Joaquin is likely the only person using ES (in the frontend) if you have questions about the specific fields.

[PGijsbers]

See also https://github.com/openml/OpenML/blob/develop/openml_OS/libraries/ElasticSearch.php (I didn't provide this initially because I didn't know where it was located, but some grepping in the code base got me there :))