8.4 release notes, http URL and config fixes

CHANGELOG.md

@@ -6,7 +6,7 @@ The format is based on [Keep a Changelog], and this project adheres to [Semantic
 
 Releases prior to 7.0 has been removed from this file to declutter search results; see the [archived copy](https://github.com/dipdup-io/dipdup/blob/8.0.0b5/CHANGELOG.md) for the full list.
 
-## [8.4.0] - 2025-05-19
+## [8.4.0] - 2025-05-20
 
 ### Added
 
@@ -22,6 +22,7 @@ Releases prior to 7.0 has been removed from this file to declutter search result
 - cli: Fixed printing help message when running commands without arguments.
 - codegen: Fixed loading ABIs from the project with no ABI datasources configured.
 - env: Skip detection of some variables if set explicitly.
+- http: Fixed merging request URL with base path.
 - project: Fixed `make image` command and default workdir.
 - project: Fixed missing codegen headers in project base.
 - substrate.node: Fixed event index field.

docs/5.advanced/2.environment-variables.md

@@ -43,8 +43,7 @@ You can also access these values as `dipdup.env` module attributes.
 You can also create a `dipdup.env` file in the current working directory. This file will be loaded automatically when you run DipDup and take precedence over other sources. This is mostly useful for internal variables.
 
 ```shell
-## Always run in debug mode
-touch dipdup.env
+## Enable debug mode for every `dipdup` command from cwd
 echo "DIPDUP_DEBUG=1" >> dipdup.env
 ```
 

docs/5.advanced/6.sqd-cloud.md

@@ -31,7 +31,7 @@ deploy:
     postgres:
     hasura:
   processor:
-    cmd: ["dipdup", "-c", "squid-cloud", "run"]
+    cmd: ["dipdup", "-C", "squid-cloud", "run"]
   init:
     cmd: ["echo", "dipdup"]
 ```

docs/7.references/2.config.md

@@ -1345,6 +1345,24 @@ description: "Config file reference"
 
 <dl class="py class">
 
+## dipdup.config.WatchdogConfig
+
+<em class="property"><span class="k"><span class="pre">class</span></span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.config.</span></span><span class="sig-name descname"><span class="pre">WatchdogConfig</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span></dt>
+<dd><p>Config for the watchdog</p>
+<dl class="field-list simple">
+<dt class="field-odd" style="color: var(--txt-primary);">Parameters<span class="colon">:</span></dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>action</strong> – Action to perform when watchdog timeout is reached</p></li>
+<li><p><strong>timeout</strong> – Watchdog timeout in seconds</p></li>
+
+<li><p><strong>kwargs</strong> (<em>Any</em>)</p></li>
+</ul>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="py class">
+
 ## dipdup.config.evm_blockvision.EvmBlockvisionDatasourceConfig
 
 <em class="property"><span class="k"><span class="pre">class</span></span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.config.evm_blockvision.</span></span><span class="sig-name descname"><span class="pre">EvmBlockvisionDatasourceConfig</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span></dt>

docs/9.release-notes/1.v8.4.md

@@ -0,0 +1,70 @@
+---
+title: "8.4"
+description: DipDup 8.4 release notes
+---
+
+<!-- markdownlint-disable no-inline-html -->
+
+# Release Notes: 8.4
+
+DipDup 8.4 introduces improvements to environment variable management, a new watchdog service for enhanced reliability, and numerous stability fixes.
+
+## Watchdog service
+
+A new watchdog service has been added to help detect and handle situations where the indexer may become unresponsive. Common causes for the indexer hanging include:
+
+- Long-running callback functions. Possible causes include infinite loops, heavy computations, or blocking I/O operations.
+- Stalled SQL transactions, which can occur due to deadlocks or long-running queries.
+- Unresponsive Websocket connections (ping-pong successful but no data received).
+
+The watchdog monitors these triggers and take action if they exceed a specified timeout. Actions can be set to `warning`, `exception`, or `ignore` for each type.
+
+Example configuration with default values:
+
+```yaml
+advanced:
+  watchdog:
+    callback:
+        action: warning
+        timeout: 10
+    transaction:
+        action: warning
+        timeout: 10
+    websocket:
+        action: warning
+        timeout: 60
+```
+
+## Managing env variables
+
+### Envvfile generation
+
+Before this release, DipDup used `.env.default` files to help users manage environment variables. These files were automatically generated on every `init` call and included in the project package. When setting up new environments, users were expected to copy the `.env.default` file to `.env` and modify it as needed. This approach had several drawbacks:
+
+- `.env.default` stubs could easily become outdated, leading to potential misconfiguration.
+- Including any envfile-alike files in the project package increased the risk of accidentally committing secrets to git.
+
+In this release, `.env.default` files have been removed. Instead, use the `config env` command to manage environment variables in a more robust and flexible way. It ensures that your environment is always up to date, allows you to resolve the current environment, chain configuration files as needed, and add internal `DIPDUP_` variables.
+
+```shell [Terminal]
+# Show variables used in the root config
+dipdup config env
+
+# Create a new envfile for `compose` environment
+dipdup -C compose config env -o .env
+
+# Add another config to the chain and include internal variables
+dipdup -C compose -C dev config env -o .env --internal
+```
+
+### `dipdup.env` file
+
+Now you can create a `dipdup.env` file in the current working directory. It will be loaded automatically when you run DipDup and take precedence over other sources. This is mostly useful for internal variables.
+
+```shell
+## Enable debug mode for every `dipdup` command from cwd
+echo "DIPDUP_DEBUG=1" >> dipdup.env
+```
+
+{{ #include 9.release-notes/_8.4_changelog.md }}
+{{ #include 9.release-notes/_footer.md }}

docs/9.release-notes/5.v7.5.md → docs/9.release-notes/6.v7.5.md

@@ -15,7 +15,7 @@ A bunch of performance improvements have been made in this release. DipDup now i
 
 The Hasura adapter now supports the `bulk` request type to apply table customizations faster and organize custom metadata files more conveniently.
 
-Finally, DipDup 6.5, the stable release branch, has reached end-of-life. 6.5.16 is the last release in this branch. Please, follow the [7.0 Migration Guide](../9.release-notes/10.v7.0.md#migration-guide) to upgrade to the latest version.
+Finally, DipDup 6.5, the stable release branch, has reached end-of-life. 6.5.16 is the last release in this branch. Please, follow the [7.0 Migration Guide](../9.release-notes/11.v7.0.md#migration-guide) to upgrade to the latest version.
 
 {{ #include 9.release-notes/_7.5_changelog.md }}
 {{ #include 9.release-notes/_footer.md }}

docs/config.rst

@@ -69,5 +69,7 @@
 .. autoclass:: dipdup.config.ReindexingAction
 .. autoclass:: dipdup.config.WatchdogTrigger
 .. autoclass:: dipdup.config.WatchdogAction
+.. autoclass:: dipdup.config.WatchdogConfig
 .. autoclass:: dipdup.config.evm_blockvision.EvmBlockvisionDatasourceConfig
 .. autoclass:: dipdup.config.ReindexingReason
+

requirements.txt

@@ -315,26 +315,26 @@ msgpack==1.1.0 \
     --hash=sha256:ad33e8400e4ec17ba782f7b9cf868977d867ed784a1f5f2ab46e7ba53b6e1e1b \
     --hash=sha256:d46cf9e3705ea9485687aa4001a76e44748b609d260af21c4ceea7f2212a501d \
     --hash=sha256:dd432ccc2c72b914e4cb77afce64aab761c1137cc698be3984eee260bcb2896e
-multidict==6.4.3 \
-    --hash=sha256:11990b5c757d956cd1db7cb140be50a63216af32cd6506329c2c59d732d802db \
-    --hash=sha256:1535cec6443bfd80d028052e9d17ba6ff8a5a3534c51d285ba56c18af97e9713 \
-    --hash=sha256:1f1c2f58f08b36f8475f3ec6f5aeb95270921d418bf18f90dffd6be5c7b0e676 \
-    --hash=sha256:26ae9ad364fc61b936fb7bf4c9d8bd53f3a5b4417142cd0be5c509d6f767e2f1 \
-    --hash=sha256:30a3ebdc068c27e9d6081fca0e2c33fdf132ecea703a72ea216b81a66860adde \
-    --hash=sha256:3ada0b058c9f213c5f95ba301f922d402ac234f1111a7d8fd70f1b99f3c281ec \
-    --hash=sha256:3b73e7227681f85d19dec46e5b881827cd354aabe46049e1a61d2f9aaa4e285a \
-    --hash=sha256:3cd06d88cb7398252284ee75c8db8e680aa0d321451132d0dba12bc995f0adcc \
-    --hash=sha256:4543d8dc6470a82fde92b035a92529317191ce993533c3c0c68f56811164ed07 \
-    --hash=sha256:59fe01ee8e2a1e8ceb3f6dbb216b09c8d9f4ef1c22c4fc825d045a147fa2ebc9 \
-    --hash=sha256:659318c6c8a85f6ecfc06b4e57529e5a78dfdd697260cc81f683492ad7e9435a \
-    --hash=sha256:66eb80dd0ab36dbd559635e62fba3083a48a252633164857a1d1684f14326427 \
-    --hash=sha256:8bd2b875f4ca2bb527fe23e318ddd509b7df163407b0fb717df229041c6df5d3 \
-    --hash=sha256:8eac0c49df91b88bf91f818e0a24c1c46f3622978e2c27035bfdca98e0e18124 \
-    --hash=sha256:b038f10e23f277153f86f95c777ba1958bcd5993194fda26a1d06fae98b2f00c \
-    --hash=sha256:c23831bdee0a2a3cf21be057b5e5326292f60472fb6c6f86392bbf0de70ba731 \
-    --hash=sha256:c2e98c840c9c8e65c0e04b40c6c5066c8632678cd50c8721fdbcd2e09f21a507 \
-    --hash=sha256:c605a2b2dc14282b580454b9b5d14ebe0668381a3a26d0ac39daa0ca115eb2ae \
-    --hash=sha256:e1eb72c741fd24d5a28242ce72bb61bc91f8451877131fa3fe930edb195f7054
+multidict==6.4.4 \
+    --hash=sha256:169c4ba7858176b797fe551d6e99040c531c775d2d57b31bcf4de6d7a669847f \
+    --hash=sha256:530d86827a2df6504526106b4c104ba19044594f8722d3e87714e847c74a0275 \
+    --hash=sha256:5e2bcda30d5009996ff439e02a9f2b5c3d64a20151d34898c000a6281faa3781 \
+    --hash=sha256:622f26ea6a7e19b7c48dd9228071f571b2fbbd57a8cd71c061e848f281550e6b \
+    --hash=sha256:69ee9e6ba214b5245031b76233dd95408a0fd57fdb019ddcc1ead4790932a8e8 \
+    --hash=sha256:7cf3bd54c56aa16fdb40028d545eaa8d051402b61533c21e84046e05513d5780 \
+    --hash=sha256:7f3d3b3c34867579ea47cbd6c1f2ce23fbfd20a273b6f9e3177e256584f1eacc \
+    --hash=sha256:87a728af265e08f96b6318ebe3c0f68b9335131f461efab2fc64cc84a44aa6ed \
+    --hash=sha256:98af87593a666f739d9dba5d0ae86e01b0e1a9cfcd2e30d2d361fbbbd1a9162d \
+    --hash=sha256:9f193eeda1857f8e8d3079a4abd258f42ef4a4bc87388452ed1e1c4d2b0c8740 \
+    --hash=sha256:a920f9cf2abdf6e493c519492d892c362007f113c94da4c239ae88429835bad1 \
+    --hash=sha256:aff4cafea2d120327d55eadd6b7f1136a8e5a0ecf6fb3b6863e8aca32cd8e50a \
+    --hash=sha256:b9eb4c59c54421a32b3273d4239865cb14ead53a606db066d7130ac80cc8ec93 \
+    --hash=sha256:bd4557071b561a8b3b6075c3ce93cf9bfb6182cb241805c3d66ced3b75eff4ac \
+    --hash=sha256:be06e73c06415199200e9a2324a11252a3d62030319919cde5e6950ffeccf72e \
+    --hash=sha256:dc388f75a1c00000824bf28b7633e40854f4127ede80512b44c3cfeeea1839a2 \
+    --hash=sha256:dc8c9736d8574b560634775ac0def6bdc1661fc63fa27ffdfc7264c565bcb4f2 \
+    --hash=sha256:ecde56ea2439b96ed8a8d826b50c57364612ddac0438c39e473fafad7ae1c23b \
+    --hash=sha256:f682c42003c7264134bfe886376299db4cc0c6cd06a3295b41b347044bcb5482
 mypy-extensions==1.1.0 \
     --hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \
     --hash=sha256:52e68efc3284861e772bbcd66823fde5ae21fd2fdb51c62a211403730b916558
@@ -539,9 +539,9 @@ ruff==0.11.10 \
 scalecodec==1.2.11 \
     --hash=sha256:99a2cdbfccdcaf22bd86b86da55a730a2855514ad2309faef4a4a93ac6cbeb8d \
     --hash=sha256:d15c94965f617caa25096f83a45f5f73031d05e6ee08d6039969f0a64fc35de1
-sentry-sdk==2.28.0 \
-    --hash=sha256:14d2b73bc93afaf2a9412490329099e6217761cbab13b6ee8bc0e82927e1504e \
-    --hash=sha256:51496e6cb3cb625b99c8e08907c67a9112360259b0ef08470e532c3ab184a232
+sentry-sdk==2.29.1 \
+    --hash=sha256:8d4a0206b95fa5fe85e5e7517ed662e3888374bdc342c00e435e10e6d831aa6d \
+    --hash=sha256:90862fe0616ded4572da6c9dadb363121a1ae49a49e21c418f0634e9d10b4c19
 six==1.17.0 \
     --hash=sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274 \
     --hash=sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81

schemas/dipdup-3.0.json

@@ -80,7 +80,7 @@
         },
         "watchdog": {
           "additionalProperties": {
-            "$ref": "#/$defs/WatchdogAction"
+            "$ref": "#/$defs/WatchdogConfig"
           },
           "description": "Mapping of watchdog triggers and actions DipDup performs.",
           "propertyNames": {
@@ -3155,6 +3155,40 @@
       "title": "WatchdogAction",
       "type": "string"
     },
+    "WatchdogConfig": {
+      "additionalProperties": false,
+      "description": "Config for the watchdog",
+      "properties": {
+        "action": {
+          "anyOf": [
+            {
+              "$ref": "#/$defs/WatchdogAction"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "description": "Action to perform when watchdog timeout is reached",
+          "title": "action"
+        },
+        "timeout": {
+          "anyOf": [
+            {
+              "type": "integer"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "description": "Watchdog timeout in seconds",
+          "title": "timeout"
+        }
+      },
+      "title": "WatchdogConfig",
+      "type": "object"
+    },
     "WatchdogTrigger": {
       "enum": [
         "callback",

src/dipdup/config/__init__.py

@@ -623,6 +623,18 @@ class SkipHistory(StrEnum):
     always = 'always'
 
 
+@dataclass(config=ConfigDict(extra='forbid', defer_build=True), kw_only=True)
+class WatchdogConfig:
+    """Config for the watchdog
+
+    :param action: Action to perform when watchdog timeout is reached
+    :param timeout: Watchdog timeout in seconds
+    """
+
+    action: WatchdogAction | None = None
+    timeout: int | None = None
+
+
 # NOTE: Should be the only place where extras are allowed
 @dataclass(config=ConfigDict(extra='allow', defer_build=True), kw_only=True)
 class AdvancedConfig:
@@ -640,7 +652,7 @@ class AdvancedConfig:
     """
 
     reindex: dict[ReindexingReason, ReindexingAction] = Field(default_factory=dict)
-    watchdog: dict[WatchdogTrigger, WatchdogAction] = Field(default_factory=dict)
+    watchdog: dict[WatchdogTrigger, WatchdogConfig] = Field(default_factory=dict)
     scheduler: dict[str, Any] | None = None
     postpone_jobs: bool = False
     early_realtime: bool = False

src/dipdup/datasources/_subsquid.py

@@ -112,7 +112,7 @@ async def _fetch_worker(self, level: int) -> DatasourceConfigT:
         worker_url = (
             await self._http.request(
                 'get',
-                f'{self._config.url}/{level}/worker',
+                f'{level}/worker',
             )
         ).decode()
 

src/dipdup/dipdup.py

@@ -92,7 +92,6 @@
 from dipdup.scheduler import SchedulerManager
 from dipdup.sys import fire_and_forget
 from dipdup.transactions import TransactionManager
-from dipdup.watchdog import DEFAULT_WATCHDOGS
 from dipdup.watchdog import watchdog
 
 if TYPE_CHECKING:
@@ -767,7 +766,6 @@ async def run(self) -> None:
                 spawn_datasources_event=spawn_datasources_event,
                 start_scheduler_event=start_scheduler_event,
                 early_realtime=advanced.early_realtime,
-                watchdog_config=advanced.watchdog,
             )
 
             if tasks:
@@ -953,7 +951,6 @@ async def _set_up_background_tasks(
         spawn_datasources_event: Event,
         start_scheduler_event: Event,
         early_realtime: bool,
-        watchdog_config: dict[Any, Any],
     ) -> None:
         index_dispatcher = self._index_dispatcher
 
@@ -976,14 +973,7 @@ def _add_task(coro: Coroutine[Any, Any, None]) -> None:
         _add_task(preload_cached_models(self._config.package))
 
         # NOTE: Watchdog
-        watchdog_config = {
-            **DEFAULT_WATCHDOGS,
-            **watchdog_config,
-        }
-        for trigger, pair in watchdog_config.items():
-            action, timeout = pair
-            watchdog.register(trigger, action, timeout)
-
+        watchdog.initialize(self._ctx.config.advanced.watchdog)
         _add_task(watchdog.run(WATCHDOG_INTERVAL))
 
     async def _spawn_datasources(self, tasks: set[Task[None]]) -> Event:

src/dipdup/http.py

@@ -215,12 +215,12 @@ async def _request(
     ) -> Any:
         """Wrapped aiohttp call with preconfigured headers and ratelimiting"""
         metrics.requests_total[self._alias] += 1
-        if not url:
-            url = self._path or '/'
-        elif url.startswith('http'):
-            url = url.replace(self._url, '').rstrip('/')
-        else:
-            url = f'{self._path.rstrip("/")}/{url}'
+        url = url or '/'
+
+        if url.startswith('http'):
+            raise InvalidRequestError(msg='URL should not start with http(s)://', url=url)
+
+        url = f'{self._path.strip("/")}/{url.strip("/")}'
 
         headers = kwargs.pop('headers', {})
         headers['User-Agent'] = self.user_agent