Skeleton of description of HTTP services and registry #33
Conversation
tsdower
left a comment
tsdower
left a comment
There was a problem hiding this comment.
Purely for the registry component:
The standardID is straightforward and makes sense and mirrors HiPS support.
Do we want the columns/filters parameters to be optional or required?
Do we care about having two different delimiters?
bsipocz
left a comment
bsipocz
left a comment
There was a problem hiding this comment.
Super quick check for the registry content, IMO it needs more.
Let me know if you want me to add text instead of just saying it needs more :)
| \label{fig:exampleCollectionPropertiesFile} | ||
| \end{figure} | ||
|
|
||
| \section{Service Capabilities} \label{sec:capabilities} |
There was a problem hiding this comment.
How do you feel about having this section (or a different section) as "HATS registration" analogue for how it's done in the hips document?
| \begin{figure} | ||
| \begin{minipage}{\linewidth} | ||
| \begin{lstlisting} | ||
| <capability standardID="ivo://ivoa.net/std/hats#hats-1.0"> |
There was a problem hiding this comment.
OK, I revert my comment above, as it should be "how about a new section for HATS registration"? As this example below misses a lot of key elements.
| hats_order = 6 | ||
| moc_sky_fraction = 0.00618 | ||
| hats_builder = hats-import v0.4.4 | ||
| hats_creation_date = 2025-03-20T03\:38UTC |
There was a problem hiding this comment.
| hats_creation_date = 2025-03-20T03\:38UTC | |
| hats_creation_date = 2025-03-20T03:38UTC |
| hats_creation_date = 2025-03-20T03\:38UTC | ||
| hats_estsize = 23137775 | ||
| hats_release_date = 2024-09-18 | ||
| hats_version = v0.1 |
There was a problem hiding this comment.
Should it be v1.0 now?
| hats_version = v0.1 | |
| hats_version = v1.0 |
| The service is not required to return the columns in the same order they were requested in the query parameters. | ||
| The service may return more columns than the user requested, e.g. the spatial index column described in Section~\ref{sec:spatialIndex}, and positional (radec) columns. |
There was a problem hiding this comment.
If I remember correctly, current lsdb-server implementation always returns all the columns, but put Null values to filtered columns, so 1) parquet files size is much smaller, but 2) the schema still matches _common_metadata
| The service is not required to return the columns in the same order they were requested in the query parameters. | ||
| The service may return more columns than the user requested, e.g. the spatial index column described in Section~\ref{sec:spatialIndex}, and positional (radec) columns. | ||
|
|
||
| \subsubsection{Row Filters} |
There was a problem hiding this comment.
There is also a question on how row and column filters work together: must client/server select all the columns we are filtering rows on, or not? If I'm not wrong, the current lsdb-server adds columns used in row filters to the output. This is different from pandas/pyarrow behavior, which supports row-filtering on columns not specified in the column list, while not returning these columns in the final table.
|
|
||
| \subsubsection{Metadata} | ||
|
|
||
| For VOParquet files, each leaf parquet file may contain the same VO metadata inside the file-level metadata. |
There was a problem hiding this comment.
Should the VOParquet note be referenced in some way?
5 participants
Adds section on registry and HTTP capabilities. Closes #31. Closes #32.
Adds new properties and file names. Closes #27. Closes #28. Closes #29.