Chore/test pyramid

.cursor/rules/pytest.mdc

@@ -10,11 +10,11 @@ These rules apply when writing unit tests.
 
 - Name test files with `test_` prefix
 - Use descriptive names that match the functionality being tested
-- Place test files in the appropriate subdirectory of `tests/`:
-    - `tests/tools/` for tests related to sub-package `pipelex.tools`
-    - `tests/pipelex/` for tests related to `pipelex`and its sub-packages
-    - `tests/pipelex/cogt/` for tests related to sub-package `pipelex.cogt`
-- More precisely, for `pipelex` and `pipelex.cogt` the async tests are placed inside subdirectories named `cogt_asynch` and `pipelex_asynch`
+- Place test files in the appropriate test category directory:
+    - `tests/unit/` - for unit tests that test individual functions/classes in isolation
+    - `tests/integration/` - for integration tests that test component interactions
+    - `tests/e2e/` - for end-to-end tests that test complete workflows
+    - `tests/test_pipelines/` - for test pipeline definitions (TOML files)
 - Fixtures are defined in conftest.py modules at different levels of the hierarchy, their scope is handled by pytest
 - Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `tests.pipelex.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.
 - Always put test inside Test classes.
@@ -55,7 +55,7 @@ class TestFooBar:
         # Test implementation
 ```
 
-Sometimes it can be convenient to access the test's name in its body, for instance to include into a job_id. To achieve that, add the argument `request: FixtureRequest` into the signature and then you can get th test name using `cast(str, request.node.originalname),  # type: ignore`. 
+Sometimes it can be convenient to access the test's name in its body, for instance to include into a job_id. To achieve that, add the argument `request: FixtureRequest` into the signature and then you can get the test name using `cast(str, request.node.originalname),  # type: ignore`. 
 
 # Pipe tests
 

.cursor/rules/standards.mdc

@@ -14,6 +14,7 @@ This document outlines the coding standards and quality control procedures that
 Before finalizing a task, you must run the following command to check for linting issues, type errors, and code quality problems:
 
 ```bash
+make fix-unused-imports
 make check
 ```
 
@@ -34,20 +35,16 @@ We have several make commands for running tests:
    ```
    Use this for quick test runs that don't require LLM or image generation.
 
-2. `make ti`: Runs all tests with these markers:
-   ```
-   inference and not imgg
-   ```
-   Use this for testing LLM functionality without image generation.
-
-3. To run specific tests:
+2. To run specific tests:
    ```bash
    make tp TEST=TestClassName
    # or
    make tp TEST=test_function_name
    ```
    It matches names, so `TEST=test_function_name` is going to run all test with the function name that STARTS with `test_function_name`.
 
+Note: never run `make ti`, `make test-inference`, `make to`, `make test-ocr`, `make tg`, or `make test-imgg`: these all use inference which is costly.
+
 ## Important Project Directories
 
 ### Pipelines Directory

Makefile

@@ -207,7 +207,7 @@ cleanall: cleanderived cleanenv cleanlibraries
 codex-tests: env
 	$(call PRINT_TITLE,"Unit testing for Codex")
 	@echo "• Running unit tests for Codex (excluding inference and codex_disabled)"
-	$(VENV_PYTEST) -n auto --exitfirst --quiet -m "(dry_runnable or not inference) and not (needs_output or pipelex_api or codex_disabled)" || [ $$? = 5 ]
+	$(VENV_PYTEST) --exitfirst -m "(dry_runnable or not inference) and not (needs_output or pipelex_api or codex_disabled)" || [ $$? = 5 ]
 
 gha-tests: env
 	$(call PRINT_TITLE,"Unit testing for github actions")
@@ -318,6 +318,27 @@ test-pipelex-api: env
 ta: test-pipelex-api
 	@echo "> done: ta = test-pipelex-api"
 
+cov: env
+	$(call PRINT_TITLE,"Unit testing with coverage")
+	@echo "• Running unit tests with coverage"
+	@if [ -n "$(TEST)" ]; then \
+		$(VENV_PYTEST) --cov=$(if $(PKG),$(PKG),pipelex) -k "$(TEST)" $(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,-v)); \
+	else \
+		$(VENV_PYTEST) --cov=$(if $(PKG),$(PKG),pipelex) $(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,-v)); \
+	fi
+
+cov-missing: env
+	$(call PRINT_TITLE,"Unit testing with coverage and missing lines")
+	@echo "• Running unit tests with coverage and missing lines"
+	@if [ -n "$(TEST)" ]; then \
+		$(VENV_PYTEST) --cov=$(if $(PKG),$(PKG),pipelex) --cov-report=term-missing -k "$(TEST)" $(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,-v)); \
+	else \
+		$(VENV_PYTEST) --cov=$(if $(PKG),$(PKG),pipelex) --cov-report=term-missing $(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,-v)); \
+	fi
+
+cm: cov-missing
+	@echo "> done: cm = cov-missing"
+
 ############################################################################################
 ############################               Linting              ############################
 ############################################################################################

docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md

@@ -133,7 +133,6 @@ Analyze the document page shown in the image and explain how it relates to the p
 2. **Fixed Multiple Outputs**: Use `nb_output = N` (where N is a positive integer) when you need exactly N outputs. For example, `nb_output = 3` will try to generate 3 results. The parameter `_nb_output` will be available in the prompt template, e.g. "Give me the names of $_nb_output flowers".
 
 3. **Variable Multiple Outputs**: Use `multiple_output = true` when you need a variable-length list where the LLM determines how many outputs to generate based on the content and context.
-| `output_multiplicity`       | string or integer   | Defines the number of outputs. Use `"list"` for a variable-length list, or an integer (e.g., `3`) for a fixed-size list.                                                       | No       |
 
 ## Examples
 

pipelex/core/stuff_factory.py

@@ -1,7 +1,7 @@
 from typing import Any, Dict, List, Optional, Tuple
 
 import shortuuid
-from pydantic import BaseModel, Field
+from pydantic import BaseModel, Field, ValidationError
 
 from pipelex.config import get_config
 from pipelex.core.concept import Concept
@@ -11,6 +11,7 @@
 from pipelex.core.stuff_content import StuffContent, StuffContentInitableFromStr
 from pipelex.exceptions import ConceptError, PipelexError
 from pipelex.hub import get_class_registry, get_required_concept
+from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error
 
 
 class StuffFactoryError(PipelexError):
@@ -148,14 +149,22 @@ def make_multiple_stuff_from_str(cls, str_stuff_and_concepts_dict: Dict[str, Tup
         return result
 
     @classmethod
-    def combine_stuffs(cls, concept_code: str, stuff_contents: Dict[str, StuffContent], name: Optional[str] = None) -> Stuff:
+    def combine_stuffs(
+        cls,
+        concept_code: str,
+        stuff_contents: Dict[str, StuffContent],
+        name: Optional[str] = None,
+    ) -> Stuff:
         """
         Combine a dictionary of stuffs into a single stuff.
         """
         the_concept = get_required_concept(concept_code=concept_code)
         the_subclass_name = the_concept.structure_class_name
         the_subclass = get_class_registry().get_required_subclass(name=the_subclass_name, base_class=StuffContent)
-        the_stuff_content = the_subclass.model_validate(obj=stuff_contents)
+        try:
+            the_stuff_content = the_subclass.model_validate(obj=stuff_contents)
+        except ValidationError as exc:
+            raise StuffFactoryError(f"Error combining stuffs: {format_pydantic_validation_error(exc=exc)}") from exc
         return cls.make_stuff(
             concept_str=concept_code,
             content=the_stuff_content,

pipelex/libraries/library_manager.py

@@ -24,6 +24,7 @@
     StaticValidationError,
 )
 from pipelex.libraries.library_config import LibraryConfig
+from pipelex.tools.class_registry_utils import ClassRegistryUtils
 from pipelex.tools.misc.file_utils import find_files_in_dir
 from pipelex.tools.misc.json_utils import deep_update
 from pipelex.tools.misc.toml_utils import load_toml_from_path
@@ -74,13 +75,13 @@ def teardown(self) -> None:
     def load_libraries(self):
         log.debug("LibraryManager loading separate libraries")
 
-        KajsonManager.get_class_registry().register_classes_in_folder(
+        ClassRegistryUtils.register_classes_in_folder(
             folder_path=LibraryConfig.loaded_pipelines_path,
         )
         library_paths = [LibraryConfig.loaded_pipelines_path]
         if runtime_manager.is_unit_testing:
             log.debug("Registering test pipeline structures for unit testing")
-            KajsonManager.get_class_registry().register_classes_in_folder(
+            ClassRegistryUtils.register_classes_in_folder(
                 folder_path=LibraryConfig.test_pipelines_path,
             )
             library_paths += [LibraryConfig.test_pipelines_path]
@@ -117,6 +118,7 @@ def _load_combo_libraries(self, library_paths: List[str]):
                 pattern="*.toml",
                 is_recursive=True,
             )
+            log.debug(f"Searching for TOML files in {libraries_path}, found '{found_file_paths}'")
             if not found_file_paths:
                 log.warning(f"No TOML files found in library path: {libraries_path}")
             toml_file_paths.extend(found_file_paths)

pipelex/pipe_operators/pipe_llm_factory.py

@@ -131,7 +131,7 @@ def make_pipe_from_blueprint(
             user_images=user_images or None,
         )
 
-        llm_settings = LLMSettingChoices(
+        llm_choices = LLMSettingChoices(
             for_text=pipe_blueprint.llm,
             for_object=pipe_blueprint.llm_to_structure,
             for_object_direct=pipe_blueprint.llm_to_structure_direct,
@@ -152,7 +152,7 @@ def make_pipe_from_blueprint(
             inputs=PipeInputSpec(root=pipe_blueprint.inputs or {}),
             output_concept_code=pipe_blueprint.output,
             pipe_llm_prompt=pipe_llm_prompt,
-            llm_choices=llm_settings,
+            llm_choices=llm_choices,
             structuring_method=pipe_blueprint.structuring_method,
             prompt_template_to_structure=pipe_blueprint.prompt_template_to_structure,
             system_prompt_to_structure=pipe_blueprint.system_prompt_to_structure,

pipelex/plugins/openai/openai_factory.py

@@ -50,8 +50,8 @@ def make_openai_client(cls, llm_platform: LLMPlatform) -> openai.AsyncClient:
                     base_url=endpoint,
                 )
             case LLMPlatform.OPENAI:
-                openai_openai_config = get_config().plugins.openai_config
-                api_key = openai_openai_config.get_api_key(secrets_provider=get_secrets_provider())
+                openai_config = get_config().plugins.openai_config
+                api_key = openai_config.get_api_key(secrets_provider=get_secrets_provider())
                 the_client = openai.AsyncOpenAI(api_key=api_key)
             case LLMPlatform.VERTEXAI:
                 vertexai_config = get_config().plugins.vertexai_config

pipelex/tools/class_registry_utils.py

@@ -0,0 +1,83 @@
+import sys
+from pathlib import Path
+from typing import Any, List, Optional, Type
+
+from pipelex.hub import get_class_registry
+from pipelex.tools.typing.module_inspector import find_classes_in_module, import_module_from_file
+
+
+class ClassRegistryUtils:
+    @classmethod
+    def register_classes_in_file(
+        cls,
+        file_path: str,
+        base_class: Optional[Type[Any]],
+        is_include_imported: bool,
+    ) -> None:
+        """Processes a Python file to find and register classes."""
+        module = import_module_from_file(file_path)
+
+        # Find classes that match criteria
+        classes_to_register = find_classes_in_module(
+            module=module,
+            base_class=base_class,
+            include_imported=is_include_imported,
+        )
+
+        # Clean up sys.modules to prevent memory leaks
+        del sys.modules[module.__name__]
+
+        get_class_registry().register_classes(classes=classes_to_register)
+
+    @classmethod
+    def register_classes_in_folder(
+        cls,
+        folder_path: str,
+        base_class: Optional[Type[Any]] = None,
+        is_recursive: bool = True,
+        is_include_imported: bool = False,
+    ) -> None:
+        """
+        Registers all classes in Python files within folders that are subclasses of base_class.
+        If base_class is None, registers all classes.
+
+        Args:
+            folder_paths: List of paths to folders containing Python files
+            base_class: Optional base class to filter registerable classes
+            recursive: Whether to search recursively in subdirectories
+            exclude_files: List of filenames to exclude
+            exclude_dirs: List of directory names to exclude
+            include_imported: Whether to include classes imported from other modules
+        """
+
+        python_files = cls.find_files_in_dir(
+            dir_path=folder_path,
+            pattern="*.py",
+            is_recursive=is_recursive,
+        )
+
+        for python_file in python_files:
+            cls.register_classes_in_file(
+                file_path=str(python_file),
+                base_class=base_class,
+                is_include_imported=is_include_imported,
+            )
+
+    @classmethod
+    def find_files_in_dir(cls, dir_path: str, pattern: str, is_recursive: bool) -> List[Path]:
+        """
+        Find files matching a pattern in a directory.
+
+        Args:
+            dir_path: Directory path to search in
+            pattern: File pattern to match (e.g. "*.py")
+            recursive: Whether to search recursively in subdirectories
+
+        Returns:
+            List of matching Path objects
+        """
+        path = Path(dir_path)
+        if is_recursive:
+            return list(path.rglob(pattern))
+        else:
+            return list(path.glob(pattern))

pipelex/tools/typing/pydantic_utils.py

@@ -27,6 +27,15 @@ def format_pydantic_validation_error(exc: ValidationError) -> str:
     type_errors = [f"{'.'.join(map(str, err['loc']))}: expected {err['type']}" for err in exc.errors() if err["type"] == "type_error"]
     value_errors = [f"{'.'.join(map(str, err['loc']))}: {err['msg']}" for err in exc.errors() if err["type"] == "value_error"]
     enum_errors = [f"{'.'.join(map(str, err['loc']))}: invalid enum value" for err in exc.errors() if err["type"] == "enum"]
+    model_type_errors: List[str] = []
+    for err in exc.errors():
+        if err["type"] == "model_type":
+            field_path = ".".join(map(str, err["loc"]))
+            # Extract expected type from context if available
+            expected_type = err.get("ctx", {}).get("class_name", "unknown model type")
+            actual_input = err.get("input", "unknown")
+            actual_type = type(actual_input).__name__ if actual_input != "unknown" else "unknown"
+            model_type_errors.append(f"{field_path}: expected {expected_type}, got {actual_type}")
 
     # Add each type of error to the message if present
     if missing_fields:
@@ -39,9 +48,11 @@ def format_pydantic_validation_error(exc: ValidationError) -> str:
         error_msg += f"\nValue errors: {value_errors}"
     if enum_errors:
         error_msg += f"\nEnum errors: {enum_errors}"
+    if model_type_errors:
+        error_msg += f"\nModel type errors: {model_type_errors}"
 
     # If none of the specific error types were found, add the raw error messages
-    if not any([missing_fields, extra_fields, type_errors, value_errors, enum_errors]):
+    if not any([missing_fields, extra_fields, type_errors, value_errors, enum_errors, model_type_errors]):
         error_msg += "\nOther validation errors:"
         for err in exc.errors():
             error_msg += f"\n{'.'.join(map(str, err['loc']))}: {err['type']}: {err['msg']}"

pyproject.toml

@@ -24,7 +24,7 @@ dependencies = [
     "instructor>=1.8.3",
     "jinja2>=3.1.4",
     "json2html>=1.3.0",
-    "kajson==0.1.6",
+    "kajson==0.2.0",
     "markdown>=3.6",
     "networkx>=3.4.2",
     "openai>=1.60.1",
@@ -71,6 +71,7 @@ dev = [
     "pytest>=8.3.3",
     "pytest-asyncio>=0.24.0",
     "pytest-cov>=6.1.1",
+    "pytest-mock>=3.14.0",
     "pytest-sugar>=1.0.0",
     "pytest-xdist>= 3.6.1",
     "ruff>=0.6.8",
@@ -217,6 +218,7 @@ markers = [
     "pipelex_api: tests that require access to the Pipelex API",
 ]
 minversion = "8.0"
+xfail_strict = true
 
 [tool.coverage.run]
 source = ["pipelex"]

tests/__init__.py

@@ -1 +1,2 @@
-
+# tests package
+# Makes tests/ importable as a proper Python package

tests/cases/__init__.py

@@ -0,0 +1,22 @@
+"""Test case constants and data definitions.
+
+This package contains pure-Python test-case definitions without test logic.
+Each module exposes only data constants that can be imported cleanly.
+"""
+
+from .documents import Article, PDFTestCases
+from .images import ImageTestCases
+from .registry import ClassRegistryTestCases, FileHelperTestCases, Fruit
+from .templates import JINJA2TestCases
+from .urls import TestURLs
+
+__all__ = [
+    "Article",
+    "PDFTestCases",
+    "ImageTestCases",
+    "TestURLs",
+    "ClassRegistryTestCases",
+    "FileHelperTestCases",
+    "Fruit",
+    "JINJA2TestCases",
+]