Release v0.10.4

This release addresses bugs associated with terrain metrics, cross sections concave hulls for sub models, and discharge bounds for sub models. The documentation has also been updated, and a new optional argument is also added to the create_fim_lib endpoint.

1. An lru_cache was added to the RasGeomText cross_sections property. the htab error fixing step needed to be moved because it was found that it was not being applied to RasGeomText classes instantiated with the from_gpkg method. This would throw off the caching by making cross_sections return an empty list for class instances generated with the from_gpkg method.
2. Areas where a ras river line intersects a cross-section at multiple locations will default to treating the first intersection as the only intersection
3. A minimum of one cross-section elevation-based terrain agreement metric is now generated even when section endpoints are lower than the thalweg
4. Cross-sections vertices where a user did not enter a value for either station or elevation will be ignored.
5. A bug that was effecting the cross section concave hull for the subset models. has been resolved In some instances the entire concave hull of the source model was being used for the subset model. This has been remedied.
6. An optional arg was added to the create_fim_lib endpoint. The new arg is cog which is a Boolean indicating whether the output geotiffs should be cloud optimized geotiffs or not. The default is False.
7. A frequently asked questions (FAQ) section has been added to the readthedocs page to provide quick responses to common ripple1d questions.
8. A bug that was incorrectly handling the NWM flows has been identified and fixed. Previously we were multiplying the high_flow_threshold by 1.2 to represent the low flow applied in ripple. This was incorrect. The low flow for ripple should be simply the high_flow_threshold. The high flow for ripple should be the 100yr from the NWM multiplied by 1.2. These default values have been updated. Additionally, the multiplier values have been exposed through the API for the conflate_model endpoint and new multipliers for the RAS flows have been exposed in the extract_submodel endpoint. Furthermore, flags have been added to give the user the option to ignore ras flows, ignore nwm flows, or take the highest and lowest of the two as the submodel flow bounds.
9. New ras_to_gpkg code has been ported from the hecstac repo.
10. A bug in flow file parsing has been addressed. See #365
11. A new concave hull method has been developed, and documentation has been added for it in the ReadTheDocs.

closes #359
closes #358
closes #336
closes #346
closes #253
closes #344
closes #366
closes #365
closes #361
closes #360

Release v0.10.3

This PR step tunes a hyperparameter in the conflation process that was causing large HEC-RAS source models to not conflate with any NWM reaches.

A central step of model conflation is identifying the relevant NWM reaches within the model domain. To do this, a valid path of NWM reaches is found that stretches from the upstream end of the HEC-RAS model to the downstream end. The valid path is found by walking several potential paths and finding one that connects upstream to downstream. A max walk length was included in earlier ripple versions to prevent process deadlocking, however, the limit was set to 100 reaches. This limit was too low for many large HEC-RAS models.

Based on a performance analysis, it was found that setting a large number (3,000) for the iteration limit would not hurt conflation speed and would allow conflation of a HEC-RAS model the full length of the Missouri River.

Release v0.10.2

This release resolves a minor bug that improves conflation at confluences and a rather major bug that was causing truncation of the profile names when the profile names exceeded 16 characters.

To resolve the truncation of profile names exceeding 16 characters, each profile name is now designated an integer value (0,1,2,3, etc.) to be used as the "new" profile name in the HEC-RAS flow file. A mapping of old-to-new profile name was then stored in the description of the flow file as a json string. During post-processing this map is retrieved and used to re-map the integer profile names back to the previous naming convention for profiles.

At some confluences, a NWM reach may intersect cross-sections from both tributary HEC-RAS reaches. To determine the "correct" reach for conflation, Ripple1D selects the reach with more coverage length. At times, the reach with more coverage would lead to the NWM reach being conflated in reverse. This release handles this situation by removing any cross-sections that would lead to reverse conflation from the set of potential HEC-RAS reaches.

If this text is confusing, see the new conflation test N.

Release v0.10.1

This release addresses several bugs that were identified in v0.10.0.

Bugfixes Addressed

Ignore terrain agreement errors

A new argument terrain_agreement_ignore_error was added to the create_ras_terrain endpoint. It's default value is false, which will maintain the current functionality. When set to true, if an error is raised in the terrain agreement calculation step, the error will be logged and no terrain agreement database will be created. The endpoint will still respond with a "terrain_agreement" key, but it will have an empty value.

Handle river_reach names more generically

While improving the conflation logic in 0.10.0, we made an assumption that the river_reach identifier in a geometry file would follow the default HEC-RAS formatting. Some contractors do not follow this formatting, so we have updated the code to more generically store river_reach names instead of forcing the default RAS formatting.

Add Model Name to extract_submodel

A required arg, model_name has been added to the extract_submodel endpoint. This enables the the user to specify the name of the model instead of inferring it from the model directory name which was not full proof.

closes #326
closes #325
closes #327

Release v0.10.0

Feature Release 0.10.0

This PR addresses several issues relating to the conflation process. Those issues are grouped in the 0.10.0 milestone, and this PR successfully addresses all of them and more. Specific changes are detailed in the subsections below.

Features Added

Check Units

The source HEC-RAS model (and thus the derived submodel) must have units of English. If it does not then an error is raised in the create_model_run_normal_depth endpoint. To do this several additional changes had to be made to the schema.

• units is now passed from the source_model.gpkg to the conflation.json during conflate_model.
• source_model_metadata is now passed from the conflation.json to the ripple1d.json during extract_submodel.
• units and steady (which is a boolean indicating if the source model was steady state) is now passed from the conflation.json to the submodel.gpkg during extract_submodel. This required adding a new metadata table to the submodel.gpkg.
• create_model_run_normal_depth reads the ripple1d.json and raises an error if units is not English.

Handle Unsteady Source Model

The handling of unsteady flow files in the source RAS model has been updated. Unsteady RAS models can now be extracted for cataloging purposes without error. However, it was decided that unsteady 1d models should not be processed through ripple1d as a steady state model and so an error is raised during create_model_run_normal_depth if the forcing file extension from the source model does not contains f indicating that it is not a steady state model.

Identify Overtopped Cross Sections

A column named xs_overtopped has been to the rating curve database which indicates if any cross sections were overtopped for any plan/profile. This should help to identify when a FIM should be scrutinized, discarded, or other decision made with regards to how it should be interpret it.

Add Model Name as arg

A required arg, model_name has been added to the conflate_model and compute_conflation_metrics endpoint. This enables the the user to specify the name of the model instead of inferring it from the model directory name which was not full proof.

Confluence Improvements

Issue #222 identified errors in conflation in confluence areas. The conflation behavior has been modified to enforce that both tributary sub models will share a downstream cross-section with the upstream cross-section of the downstream sub model.

Eclipsed Reach Identification

An eclipsed reach is defined as a NWM reach that has upstream and downstream NWM reaches conflated but with no source model cross-sections intersecting itself. In previous versions of the code, ripple was both under- and over-identifying eclipsed reaches (see #311 and #316). The code has been improved to properly scan for eclipsed NWM reaches.

Conflating NWM Reaches With Low Source Model Overlap

Issues #218 and #227 detail edge cases near first-order NWM reaches. The conflation process has been updated to allow for a) scanning for NWM that intersect a small portion of the source model and b) expanding the sub model upstream cross-section above the NWM terminus if an additional cross-section is available.

Documentation and testing

13 new tests were added to prevent code regression as the conflation module was updated. These test cases also serve as examples of how conflation "should" behave. As such, graphics were generated for each test and have been added to the conflate_model endpoint page.

closes #186
closes #303
closes #123
closes #176
closes #222
closes #311
closes #227
closes #218
closes #316

Release v0.8.3

Bugfix Release 0.8.3

Summary

This release of ripple1d fixes several bugs that were identified in a large production run. Those bugs were detailed and summarized in #268.

Bug Fixes

• When a new terrain is downloaded but contains only nodata values, NullTerrainError is raised, and the endpoint process will fail.
• When a new terrain is downloaded but the HEC-RAS CreateTerrain subprocess fails silently, the output is checked, and a RasTerrainFailure error is raised. Additionally, bad_projection_test.py has been added to test that the error will raise.
• When a source model is conflated such that the most downstream XS for a NWM reach a has higher river station than the most upstream reach, a BadConflation error is raised.
• When sub model extraction is attempted on a reach that would have one or zero cross-sections but is not tagged as eclipsed, a SingleXSModel error is raised. The endpoint response was also cleaned up such that eclipsed reaches no longer return a path to a nonexistent 123.ripple1d.json. Logging has been improved for all these cases.
• flow indexing in the create_flow_depth_combinations function has been improved to avoid rounding values. A test has been added for one of the models that was causing problems.
• Handling of interpolated cross-sections has been improved such that they may have sub models created, may be run, and may be used in the create FIM lib steps. A new test model -- winooski -- has been added with interpolated cross-sections.
• Null (blank) downstream reach values are now loaded as 0. This is consistent with how HEC-RAS handles them. The new winooski model contains null downstream reach length at the most downstream cross-section.

Closes #281
Closes #282
Closes #291
Closes #292
Closes #295
Closes #298

Release v0.8.1

Re-release 0.8.1

Important

Version 0.8.0 was released without having incorporated bugfixes added after v0.8rc1. This update adds in the necessary bugfixes to reflect the most up-to-date version of the software. The text below is a copy of the 0.8.0 release notes.

Summary

This release adds a suite of diagnostic metrics on model quality and terrain agreement to the ripple1d outputs. Model quality metrics reflect how well a HEC-RAS model conforms to standards of practice for developing HEC-RAS models and provide easy access to many model attributes that would be checked during an engineering model QC. Terrain agreement metrics quantify how well the terrain generated in the create_ras_terrain endpoint matches the source model terrain. Access to these metrics give ripple1d users actionable information to inform which HEC-RAS models should be preferred when models overlap. Furthermore, elevation-specific terrain agreement metrics can be used to alert users when a FIM product substantially disagrees with a source model geometry (indicative of lower FIM quality).

Additional features have been added to handle HEC-RAS warnings about hydraulic table start points and cross-section naming.

Features Added

Hydraulic Table Start Elevation

When a hydraulic table for a cross-section has a minimum elevation below the section invert, HEC-RAS generates a pop up warning as demonstrated in https://github.com/Dewberry/ripple1d/issues/249. In the warning, the software specifies that any subsequent saves to the geometry file will automatically adjust the hydraulic table minimum elevation to be above the section invert. To mimic this functionality, a new function call has been added to the RasGeomText constructor. The call will check for any occurrences of the htab issue, adjust the minimum elevation, and overwrite the geometry file with the updated value.

HEC-RAS implements some convoluted process to update both the htab start value and the increments. We elected to simply update the htab start elevation to the cross-section invert and preserve the original engineer-determined increments.

Sub Model Cross-Section Names

In earlier versions of ripple1d, the software attempted to carry cross-section names from source model to sub model. When that naming would have resulted in decreasing values in the upstream direction, the stationing was adjusted to preserve increasing values in the names. The renaming process led to a couple of bugs in downstream processes (ex #225 and #266). Since cross-section names do not affect hydraulic computation, the software will now rename all sub model cross-sections to an autoincremented value (ex 1, 2, 3, etc).

Additionally, the extract_submodel endpoint has been refactored to fix a potential bug in the junction walking step. For cases where more than one junction separated the u/s and d/s cross-sections or cases where the junctions were not reported from upstream to downstream in the HEC-RAS geometry file, the extract_submodel endpoint would only grab a single intermediate reach. No examples of this behavior have been found in production, but the bug was tested in development with a testing HEC-RAS model.

Terrain Agreement Metrics

A new step has been added to the create_ras_terrain endpoint that will generate a json of terrain agreement metrics. The json is saved to a file with suffix ".terrain_agreement.json" that is saved to the sub model directory. The JSON contains a suite of metrics that have been documented in the Technical Summary of the readthedocs page (https://ripple1d.readthedocs.io/en/latest/tech_summary.html) and the create_ras_terrain endpoint page (https://ripple1d.readthedocs.io/en/latest/endpoints/create_ras_terrain.html).

Model Quality Metrics

A new step has been added to the extract source model endpoint which extracts source model information to determine the quality of the source model. These metrics are populated in the cross section layer located in the source model geopackage layer for every cross section. The submodel will inherit these metrics during the subset gpkg phase.

Bug Fixes

• A feature was added in v0.7.0 to allow html rendering of the jobs endpoint, however, the endpoint was not updated to accept a flag toggling between json and html responses. Additionally, an oversight in the project manifest meant that the html and css templates for the jobs page were not transferred to ripple1d when installed via pip. This has been resolved so that the jobs endpoint returns a json response by default, and an html response will be returned when the key-value pair f=html is in the query string.
• readthedocs was failing to build our documentation on their server. The docs folder has been updated such that it will build via their service. Additionally, the changelog will autogenerate now. Some old windows dependencies and imports have been removed since they are not used to run HEC-RAS any more and needed unnecessary logic to avoid breaking the readthedocs build service.

Closes #118
Closes #207
Closes #208
Closes #209
Closes #225
Closes #249
Closes #266
Closes #267
Closes #268
Closes #269
Closes #270

Release v0.8.0

Feature Release 0.8.0

Summary

This release adds a suite of diagnostic metrics on model quality and terrain agreement to the ripple1d outputs. Model quality metrics reflect how well a HEC-RAS model conforms to standards of practice for developing HEC-RAS models and provide easy access to many model attributes that would be checked during an engineering model QC. Terrain agreement metrics quantify how well the terrain generated in the create_ras_terrain endpoint matches the source model terrain. Access to these metrics give ripple1d users actionable information to inform which HEC-RAS models should be preferred when models overlap. Furthermore, elevation-specific terrain agreement metrics can be used to alert users when a FIM product substantially disagrees with a source model geometry (indicative of lower FIM quality).

Additional features have been added to handle HEC-RAS warnings about hydraulic table start points and cross-section naming.

Features Added

Hydraulic Table Start Elevation

When a hydraulic table for a cross-section has a minimum elevation below the section invert, HEC-RAS generates a pop up warning as demonstrated in https://github.com/Dewberry/ripple1d/issues/249. In the warning, the software specifies that any subsequent saves to the geometry file will automatically adjust the hydraulic table minimum elevation to be above the section invert. To mimic this functionality, a new function call has been added to the RasGeomText constructor. The call will check for any occurrences of the htab issue, adjust the minimum elevation, and overwrite the geometry file with the updated value.

HEC-RAS implements some convoluted process to update both the htab start value and the increments. We elected to simply update the htab start elevation to the cross-section invert and preserve the original engineer-determined increments.

Sub Model Cross-Section Names

In earlier versions of ripple1d, the software attempted to carry cross-section names from source model to sub model. When that naming would have resulted in decreasing values in the upstream direction, the stationing was adjusted to preserve increasing values in the names. The renaming process led to a couple of bugs in downstream processes (ex #225 and #266). Since cross-section names do not affect hydraulic computation, the software will now rename all sub model cross-sections to an autoincremented value (ex 1, 2, 3, etc).

Additionally, the extract_submodel endpoint has been refactored to fix a potential bug in the junction walking step. For cases where more than one junction separated the u/s and d/s cross-sections or cases where the junctions were not reported from upstream to downstream in the HEC-RAS geometry file, the extract_submodel endpoint would only grab a single intermediate reach. No examples of this behavior have been found in production, but the bug was tested in development with a testing HEC-RAS model.

Terrain Agreement Metrics

A new step has been added to the create_ras_terrain endpoint that will generate a json of terrain agreement metrics. The json is saved to a file with suffix ".terrain_agreement.json" that is saved to the sub model directory. The JSON contains a suite of metrics that have been documented in the Technical Summary of the readthedocs page (https://ripple1d.readthedocs.io/en/latest/tech_summary.html) and the create_ras_terrain endpoint page (https://ripple1d.readthedocs.io/en/latest/endpoints/create_ras_terrain.html).

Model Quality Metrics

A new step has been added to the extract source model endpoint which extracts source model information to determine the quality of the source model. These metrics are populated in the cross section layer located in the source model geopackage layer for every cross section. The submodel will inherit these metrics during the subset gpkg phase.

Bug Fixes

• A feature was added in v0.7.0 to allow html rendering of the jobs endpoint, however, the endpoint was not updated to accept a flag toggling between json and html responses. Additionally, an oversight in the project manifest meant that the html and css templates for the jobs page were not transferred to ripple1d when installed via pip. This has been resolved so that the jobs endpoint returns a json response by default, and an html response will be returned when the key-value pair f=html is in the query string.
• readthedocs was failing to build our documentation on their server. The docs folder has been updated such that it will build via their service. Additionally, the changelog will autogenerate now. Some old windows dependencies and imports have been removed since they are not used to run HEC-RAS any more and needed unnecessary logic to avoid breaking the readthedocs build service.

Closes #118
Closes #207
Closes #208
Closes #209
Closes #225
Closes #249
Closes #266
Closes #267
Closes #268
Closes #269
Closes #270

Release v0.8rc1

Feature Release 0.8.0

Summary

This release adds a suite of diagnostic metrics on model quality and terrain agreement to the ripple1d outputs. Model quality metrics reflect how well a HEC-RAS model conforms to standards of practice for developing HEC-RAS models and provide easy access to many model attributes that would be checked during an engineering model QC. Terrain agreement metrics quantify how well the terrain generated in the create_ras_terrain endpoint matches the source model terrain. Access to these metrics give ripple1d users actionable information to inform which HEC-RAS models should be preferred when models overlap. Furthermore, elevation-specific terrain agreement metrics can be used to alert users when a FIM product substantially disagrees with a source model geometry (indicative of lower FIM quality).

Additional features have been added to handle HEC-RAS warnings about hydraulic table start points and cross-section naming.

Features Added

Hydraulic Table Start Elevation

When a hydraulic table for a cross-section has a minimum elevation below the section invert, HEC-RAS generates a pop up warning as demonstrated in https://github.com/Dewberry/ripple1d/issues/249. In the warning, the software specifies that any subsequent saves to the geometry file will automatically adjust the hydraulic table minimum elevation to be above the section invert. To mimic this functionality, a new function call has been added to the RasGeomText constructor. The call will check for any occurrences of the htab issue, adjust the minimum elevation, and overwrite the geometry file with the updated value.

HEC-RAS implements some convoluted process to update both the htab start value and the increments. We elected to simply update the htab start elevation to the cross-section invert and preserve the original engineer-determined increments.

Sub Model Cross-Section Names

In earlier versions of ripple1d, the software attempted to carry cross-section names from source model to sub model. When that naming would have resulted in decreasing values in the upstream direction, the stationing was adjusted to preserve increasing values in the names. The renaming process led to a couple of bugs in downstream processes (ex #225 and #266). Since cross-section names do not affect hydraulic computation, the software will now rename all sub model cross-sections to an autoincremented value (ex 1, 2, 3, etc).

Additionally, the extract_submodel endpoint has been refactored to fix a potential bug in the junction walking step. For cases where more than one junction separated the u/s and d/s cross-sections or cases where the junctions were not reported from upstream to downstream in the HEC-RAS geometry file, the extract_submodel endpoint would only grab a single intermediate reach. No examples of this behavior have been found in production, but the bug was tested in development with a testing HEC-RAS model.

Terrain Agreement Metrics

A new step has been added to the create_ras_terrain endpoint that will generate a json of terrain agreement metrics. The json is saved to a file with suffix ".terrain_agreement.json" that is saved to the sub model directory. The JSON contains a suite of metrics that have been documented in the Technical Summary of the readthedocs page (https://ripple1d.readthedocs.io/en/latest/tech_summary.html) and the create_ras_terrain endpoint page (https://ripple1d.readthedocs.io/en/latest/endpoints/create_ras_terrain.html).

Model Quality Metrics

A new step has been added to the extract source model endpoint which extracts source model information to determine the quality of the source model. These metrics are populated in the cross section layer located in the source model geopackage layer for every cross section. The submodel will inherit these metrics during the subset gpkg phase.

Bug Fixes

• A feature was added in v0.7.0 to allow html rendering of the jobs endpoint, however, the endpoint was not updated to accept a flag toggling between json and html responses. Additionally, an oversight in the project manifest meant that the html and css templates for the jobs page were not transferred to ripple1d when installed via pip. This has been resolved so that the jobs endpoint returns a json response by default, and an html response will be returned when the key-value pair f=html is in the query string.
• readthedocs was failing to build our documentation on their server. The docs folder has been updated such that it will build via their service. Additionally, the changelog will autogenerate now. Some old windows dependencies and imports have been removed since they are not used to run HEC-RAS any more and needed unnecessary logic to avoid breaking the readthedocs build service.

Closes #118
Closes #207
Closes #208
Closes #209
Closes #225
Closes #249
Closes #266
Closes #267
Closes #268
Closes #269
Closes #270

Release v0.7.0

Feature Release 0.7.0

Users Changelog

This release of ripple1d attempts to speed up the known water surface elevation (kwse) computes by parallelizing the creation of the depth grids. To do this, all kwse runs are ran without producing depth grids. From these initial kwse runs the rating curve database are created and used to inform the boundary condition for the next reach upstream. Once all of the boundary conditions are known, a second kwse run is ran to produce the depth grids in parallel. To make this happen some changes were necessary to the endpoints.

In addition to adding/modifying the endpoints, this release of ripple1d makes significant updates to both ripple1d execution and logging. This includes running each endpoint as a separate subprocess which allows the user to have the ability to dismiss running jobs. This should be handy for when jobs appear to have hung up. Dismissing these hung up jobs will free up cpu for new jobs.

Features Added

Write depth grids argument

A new argument boolean "write_depth_grids" was added to the "run_incremental_normal_depth" and "run_known_wse" endpoints. This allows the user to specify whether ripple1d should compute raw RAS depth grids or not.

Create rating curves database

A new endpoint called "create_rating_curves_db" was added. This endpoint is a post processing step that creates a rating curve database from RAS plan hdf results. This endpoint only requires 2 args: "submodel_directory" and "plans. The location of the rating curve database is inferred from the submodel directory. It will be located directly in the submodel directory and will be named as the network reach name; e.g., "2820002/2820002.db".

The "create_rating_curves_db" endpoint checks for the presence of the depth grid for each profile and records its existence or absence along with the plan suffix in the database table. The columns are "plan_suffix" and "map_exist" and are of type string and boolean, respectively.

Create FIM library update

The "create_fim_lib" endpoint no longer produces the rating curve database.

Update to the Ripple1d workflow

The user should take care with the args when calling the new/modified endpoints; specifically the plan names. The recommended order for calling these endpoints is:

1. run_known_wse : with "write_depth_grids" set to false and "plan_suffix" set to ["ikwse"]
2. create_rating_curves_db: with "plans" set to ["ikwse"]
3. run_known_wse: with "write_depth_grids" set to true and "plan_suffix" set to ["nd","kwse"]
4. create_rating_curves_db: with "plans" set to ["nd","kwse"]
5. create_fim_lib: with "plans" set to ["nd","kwse"]

Subprocess encapsulation

The execution of all endpoints are now encapsulated within subprocesses

Logs and database

• Huey and flask logs have been combined into a single log file (server-logs.jsonld).
• Huey.db has been renamed to jobs.db
• The process id for each endpoint are now tracked in a p_id field of the task_status table in jobs.db
• A huey_status field has been added to the task_status table. This field tracks the execution status of the endpoint subprocess.
• A new table called task_logs has been added to jobs.db. This table contains stdout, stderr, and results stemming from endpoint subprocesses.
• A proof of concept graphical (html-based) job status page has been added

Closes #187
Closes #188
Closes #237
Closes #238
Closes #239
Closes #240