dipdup.env loading, cli and env fixes

.gitignore

@@ -27,7 +27,6 @@
 !**/*.yml
 !**/*.sh
 !**/*.j2
-!**/*.env.default
 
 # Docs
 !docs/**

.vscode/settings.json

@@ -5,7 +5,4 @@
             "tests/configs/*.y[a]ml"
         ],
     },
-    "files.associations": {
-        "*.env.default": "shellscript"
-    }
 }

CHANGELOG.md

@@ -6,19 +6,30 @@ The format is based on [Keep a Changelog], and this project adheres to [Semantic
 
 Releases prior to 7.0 has been removed from this file to declutter search results; see the [archived copy](https://github.com/dipdup-io/dipdup/blob/8.0.0b5/CHANGELOG.md) for the full list.
 
-## [8.3.4] - 2025-05-07
+## [8.3.4] - 2025-05-19
 
 ### Added
 
 - context: Added configurable watchdog service to notify about long-running callbacks and transactions.
 - cli: Added loading env-file `dipdup.env` if presented in the current directory.
+- cli: Added `-h` shorthand for `--help` option in all commands.
+- cli: Added `--no-types` option to `init` command to skip generating type classes.
 
 ### Fixed
 
+- cli: Fixed `config env -i` command output.
+- cli: Fixed discovering package path.
+- cli: Fixed printing help message when running commands without arguments.
 - codegen: Fixed loading ABIs from the project with no ABI datasources configured.
+- env: Skip detection of some variables if set explicitly.
 - project: Fixed `make image` command and default workdir.
+- project: Fixed missing codegen headers in project base.
 - substrate.node: Fixed event index field.
 
+### Removed
+
+- project: Do not generate and track `.env.default` files on init. Use `config env` command for new environments.
+
 ## [8.3.3] - 2025-04-29
 
 ### Added

Makefile

@@ -54,7 +54,7 @@ docs:           ## Build docs
 	python scripts/docs.py check-links --source docs
 	python scripts/docs.py dump-references
 	python scripts/docs.py dump-demos
-	python scripts/docs.py dump-metrics
+	python scripts/docs.py dump-ref-tables
 	python scripts/docs.py dump-jsonschema
 	python scripts/docs.py merge-changelog
 	python scripts/docs.py markdownlint
@@ -87,7 +87,7 @@ demos:          ## Recreate demo projects from templates
 	DIPDUP_NO_SYMLINK=1 python scripts/demos.py init ${DEMO}
 
 demos_refresh:  ## Run `init --force` in all demo projects
-	for demo in `ls src | grep demo | grep -v etherlink`; do cd src/$$demo && DIPDUP_NO_SYMLINK=1 dipdup init --force && cd ../..; done
+	for demo in `ls src | grep demo | grep -v etherlink`; do DIPDUP_NO_SYMLINK=1 dipdup -c src/$$demo init --force --no-types; done
 
 before_release: ## Prepare for a new release after updating version in pyproject.toml
 	make format lint update demos test docs

docs/0.quickstart-evm.md

@@ -34,13 +34,10 @@ dipdup_indexer [.]
 │   ├── dipdup.swarm.yaml
 │   └── replay.yaml
 ├── deploy
-│   ├── .env.default
 │   ├── Dockerfile
 │   ├── compose.sqlite.yaml
 │   ├── compose.swarm.yaml
 │   ├── compose.yaml
-│   ├── sqlite.env.default
-│   └── swarm.env.default
 ├── graphql
 ├── handlers
 │   └── on_transfer.py

docs/0.quickstart-starknet.md

@@ -34,13 +34,10 @@ dipdup_indexer [.]
 │   ├── dipdup.swarm.yaml
 │   └── replay.yaml
 ├── deploy
-│   ├── .env.default
 │   ├── Dockerfile
 │   ├── compose.sqlite.yaml
 │   ├── compose.swarm.yaml
 │   ├── compose.yaml
-│   ├── sqlite.env.default
-│   └── swarm.env.default
 ├── graphql
 ├── handlers
 │   └── on_transfer.py

docs/0.quickstart-substrate.md

@@ -34,13 +34,10 @@ dipdup_indexer [.]
 │   ├── dipdup.swarm.yaml
 │   └── replay.yaml
 ├── deploy
-│   ├── .env.default
 │   ├── Dockerfile
 │   ├── compose.sqlite.yaml
 │   ├── compose.swarm.yaml
 │   ├── compose.yaml
-│   ├── sqlite.env.default
-│   └── swarm.env.default
 ├── graphql
 ├── handlers
 │   ├── batch.py

docs/0.quickstart-tezos.md

@@ -31,13 +31,10 @@ dipdup_indexer [.]
 │   ├── dipdup.swarm.yaml
 │   └── replay.yaml
 ├── deploy
-│   ├── .env.default
 │   ├── Dockerfile
 │   ├── compose.sqlite.yaml
 │   ├── compose.swarm.yaml
 │   ├── compose.yaml
-│   ├── sqlite.env.default
-│   └── swarm.env.default
 ├── graphql
 ├── handlers
 │   ├── on_balance_update.py

docs/1.getting-started/1.installation.md

@@ -9,7 +9,7 @@ This page covers the installation of DipDup in different environments.
 
 ## Host requirements
 
-A Linux/macOS environment with Python 3.12 installed is required to use DipDup. Also, you need a basic Python environment for the install script, `python3.12 -m ensurepip`. Apple silicon is supported. Other UNIX-like systems should work but are not supported officially. If you use Windows, please use [WSL](https://docs.microsoft.com/en-us/windows/wsl/about) or Docker.
+A Linux/macOS environment with Python 3.12 installed is required to use DipDup. Apple silicon is supported. Other UNIX-like systems should work but are not supported officially. If you use Windows, please use [WSL](https://docs.microsoft.com/en-us/windows/wsl/about) or Docker.
 
 Minimum hardware requirements are 256 MB RAM, 1 CPU core, and some disk space for the database. RAM requirements increase with the number of indexes.
 

docs/1.getting-started/2.core-concepts.md

@@ -11,15 +11,16 @@ Before diving into development, let's explore the fundamental concepts of DipDup
 
 DipDup is a **Python SDK** for building customized blockchain indexers. These indexers are off-chain services that collect, process, and store data from blockchain networks in a structured database format.
 
-Each DipDup indexer consists of two main components: a **YAML configuration file** and a **Python package**. The configuration file defines which contracts to index, what data to extract, and where to store the results. DipDup's configuration system supports advanced features like templates, environment variable substitution, and multi-file merging, enabling fully declarative indexer definitions. If you're familiar with The Graph, you'll notice similarities to Subgraph manifests.
+Each DipDup indexer consists of two main components: a **YAML [configuration file](./3.config.md)** and a **Python package**. The configuration file defines which contracts to index, what data to extract, and where to store the results. DipDup's configuration system supports advanced features like templates, environment variable substitution, and multi-file merging, enabling fully declarative indexer definitions. If you're familiar with The Graph, you'll notice similarities to Subgraph manifests.
 
 An **index** represents a collection of contracts and processing rules treated as a single entity. Your configuration can include multiple indices that operate in parallel, though they cannot share data as execution order isn't guaranteed.
 
-The Python package contains several components:
+The **Python package** is where your project's custom logic resides. It typically includes:
 
-- **ORM models** that define your domain-specific data structures
-- **Handlers** (callbacks) that implement business logic for transforming blockchain data into your models
-- Optional components like typeclasses, SQL scripts, GraaphQL queries etc. to extend DipDup's functionality
+- **[ORM models](./6.models.md)**: Python classes, usually in a `models.py` file or a `models` directory, that define the structure of your database tables. These models inherit from `dipdup.models.Model` and use `dipdup.fields` to define columns.
+- **[Handlers](./9.handlers.md)**: Python functions or methods, often organized in `handlers.py` or a `handlers` directory. These are callbacks that DipDup executes in response to specific blockchain events or data, implementing your business logic to process data and save it using your ORM models.
+- **[Hooks](./10.hooks.md)**: Special callback functions (e.g., `on_reindex`, `on_rollback`) located in `hooks.py` or a `hooks` directory, allowing you to execute custom logic at different stages of the indexing lifecycle.
+- Optional components like custom typeclasses, SQL scripts (in a `sql` directory), and GraphQL queries (in a `graphql` directory) to further customize and extend DipDup's functionality.
 
 The result is a service that populates your database with indexed blockchain data, which you can then use to build custom API backends or integrate with existing solutions. DipDup provides built-in **Hasura GraphQL Engine** integration, exposing your indexed data through REST and GraphQL APIs with zero configuration. Alternatively, you can use other API engines like PostgREST or develop your own.
 
@@ -44,6 +45,6 @@ Updates are applied **atomically** on a block-by-block basis, ensuring data inte
 
 ## Handling chain reorganizations
 
-Chain reorganizations occur when some blocks (including all their operations) are **rolled back** in favor of others with higher fitness. DipDup processes these reorganization events by restoring a previous database state. You can also implement custom rollback logic by modifying the `on_index_rollback` system hook in Python.
+Chain reorganizations occur when some blocks (including all their operations) are **rolled back** in favor of others with higher fitness. DipDup processes these reorganization events by restoring a previous database state. You can also implement custom rollback logic by modifying the `on_index_rollback` [system hook](./10.hooks.md) in Python.
 
 Keep in mind, that automatic database rollback works only with ORM models. If you use raw SQL queries, you need to handle reorganizations manually.

docs/1.getting-started/3.config.md

@@ -47,8 +47,6 @@ Use `config export`{lang="shell"} and `config env`{lang="shell"} commands to che
 
 ## Environment variables
 
-_For the list of variables to configure DipDup see [this page](../5.advanced/2.environment-variables.md)._
-
 DipDup supports compose-style variable expansion with an optional default value. Use this feature to store sensitive data outside of the configuration file and make your app fully declarative. If a required variable is not set, DipDup will fail with an error. You can use these placeholders anywhere throughout the configuration file.
 
 ```yaml [dipdup.yaml]
@@ -63,14 +61,7 @@ There are multiple ways to pass environment variables to DipDup:
 - Export them in the shell before running DipDup
 - Create the env file and pass it to DipDup with the `-e` CLI option
 
-For every config file in the `deploy` project directory, DipDup will create a corresponding `.env.default` file with all the variables used in the config. Copy it, remove the `.default` suffix and fill in the values.
-
-```sh [deploy/.env.default]
-POSTGRES_HOST=localhost
-POSTGRES_PASSWORD=
-```
-
-You can use `dipdup -e .env config export --unsafe`{lang="shell"} command to ensure that env variables resolve as expected, but avoid sharing the output with third parties.
+See [Environment Variables](../5.advanced/2.environment-variables.md) page for details.
 
 ## Contract typenames
 

docs/1.getting-started/5.database.md

@@ -11,7 +11,7 @@ DipDup officially supports SQLite, PostgreSQL and TimescaleDB as a database engi
 
 |                      | SQLite            | PostgreSQL    | TimescaleDB             |
 |:-------------------- |:----------------- |:------------- |:----------------------- |
-| Recommended versions | latest            | 14, 15        | 14, 15                  |
+| Recommended versions | latest            | latest        | latest                  |
 | Best application     | early development | general usage | working with timeseries |
 | SQL scripts          | 🟢                | 🟢            | 🟢                      |
 | Immune tables        | ⚠️                | 🟢            | 🟢                      |

docs/1.getting-started/8.indexes.md

@@ -5,9 +5,9 @@ description: "Index is a primary DipDup entity connecting the inventory and data
 
 # Indexes
 
-Index is a primary DipDup entity connecting the inventory and data handling rules.
+An **index** is a core DipDup entity that connects your data inventory with the rules for handling blockchain data. Indexes define *what* to track (contracts, events, operations, etc.), *where* to get the data (datasource), and *how* to process it (handlers). They are the main building blocks for structuring your indexing logic in DipDup.
 
-Multiple indexes are available for different workloads. Every index is linked to a specific datasource and provides a set of handlers for different kinds of data. Use this table to choose the right index for the task:
+Multiple index kinds are available for different blockchains and workloads. Each index is linked to a specific datasource and provides a set of handlers for different kinds of data. Use the table below to choose the right index for your task:
 
 | kind                                                                         | blockchain       | datasources | indexed data                |
 | ---------------------------------------------------------------------------- | ---------------- | ----------- | --------------------------- |
@@ -23,9 +23,11 @@ Multiple indexes are available for different workloads. Every index is linked to
 | [tezos.token_balances](../2.indexes/10.tezos_token_balances.md)              | ꜩ Tezos          | `tezos`     | TZIP-12/16 token balances   |
 | [tezos.token_transfers](../2.indexes/11.tezos_token_transfers.md)            | ꜩ Tezos          | `tezos`     | TZIP-12/16 token transfers  |
 
-Indexes can join multiple contracts considered as a single application. Also, contracts can be used by multiple indexes of any kind, but make sure that they are independent of each other and that indexed data don't overlap.
+Indexes can join multiple contracts considered as a single application. Contracts can be used by multiple indexes of any kind, but make sure that they are independent of each other and that indexed data don't overlap.
 
-Handler is a callback function, called when new data has arrived from the datasource. The handler receives the data item as an argument and can perform any actions, e.g., store data in the database, send a notification, or call an external API.
+A **handler** is a callback function triggered when new data arrives from the datasource. Handlers receive the data item as an argument and can perform any action, such as storing data in the database, sending notifications, or calling external APIs.
+
+Handlers are defined in your config and linked to specific patterns (e.g., entrypoints, events, or operations).
 
 ## Using templates
 
@@ -47,7 +49,7 @@ indexes:
       - tzkt_mainnet
 ```
 
-You can also spawn indexes from templates in runtime; see [Spawning in runtime](../1.getting-started/8.indexes.md#spawning-in-runtime).
+You can also spawn indexes from templates at runtime; see [Spawning in runtime](../1.getting-started/8.indexes.md#spawning-in-runtime).
 
 ## Limiting scope
 
@@ -75,7 +77,7 @@ templates:
     contracts:
       - <contract>
     handlers:
-      - callback: callback
+      - callback: on_call
         pattern:
           - destination: <contract>
             entrypoint: call
@@ -105,7 +107,7 @@ An index created from a template must have a value for each placeholder; the exc
 
 ## Spawning in runtime
 
-You can also spawn indexes from templates in runtime. To begin with, you need to define index templates in the top-level `templates` section of the config. Then call `ctx.add_contract` and `ctx.add_index` methods from any user callback. To achieve the same effect as in snippets from previous paragraph, you can use the following code:
+DipDup allows you to spawn indexes from templates dynamically at runtime. To do this, define index templates in the top-level `templates` section of your config. Then, from any user callback, call `ctx.add_contract` and `ctx.add_index`:
 
 ```python
 await ctx.add_contract(