Skip to content

Conversation

@dabla
Copy link
Contributor

@dabla dabla commented Jan 13, 2026

Was generative AI tooling used to co-author this PR?

  • Yes (please specify the tool below)

This PR refactors HttpAsyncHook to natively support session-based async HTTP usage, making it easier and safer to perform multiple HTTP requests within a single aiohttp.ClientSession.

As part of this refactor, LivyAsyncHook (which extends HttpAsyncHook) is simplified to reuse shared logic, removing duplicated code and avoiding internal state mutation during request execution.

Before this change, users who wanted to reuse an aiohttp.ClientSession had to manage the session lifecycle themselves and pass it into HttpAsyncHook.run(). This resulted in boilerplate-heavy task code and made it harder to apply hook-level configuration consistently.

Instead of having to do this:

    @task(
        retries=3,
        retry_delay=timedelta(seconds=60),
        show_return_value_in_logs=True,
    )
    async def fetch_forms(project_id: int):
        from aiohttp import ClientSession
        from functools import partial

        from airflow.providers.http.hooks.http import HttpAsyncHook

        auth_type = partial(BearerTokenAuth, url="https://localhost/v1/sessions")
        hook = HttpAsyncHook(http_conn_id="http_conn_id", auth_type=auth_type, method="GET")

        async with ClientSession() as session:
            response = await hook.run(
                session=session,
                endpoint=f"v1/projects/{project_id}/forms",
            )
            return await response.json()

You will be able to do this instead as the HttpAsyncHook now exposes a session-based API that encapsulates session creation, configuration, authentication, and retry logic:

    @task(
        retries=3,
        retry_delay=timedelta(seconds=60),
        show_return_value_in_logs=True,
    )
    async def fetch_forms(project_id: int):
        from functools import partial

        from airflow.providers.http.hooks.http import HttpAsyncHook

        auth_type = partial(
            BearerTokenAuth, url="https://localhost/v1/sessions"
        )
        hook = HttpAsyncHook(
            http_conn_id="http.odk.acc", auth_type=auth_type, method="GET"
        )

        response = await hook.run(
            endpoint=f"v1/projects/{project_id}/forms",
        )
        return await response.json()

        # or if multiple requests need to be done with the same session:
        async with hook.session() as session:
            response_1 = await session.run(
                endpoint=f"v1/projects/{project_id}/forms",
            )
            return await response_1.json()
            
            response_2 = await session.run(
                endpoint=f"v1/projects/{project_id}/forms",
            )
            return await response_2.json()

This approach:

  • Eliminates manual aiohttp.ClientSession management in tasks (no more boilerplate code needed).
  • Makes multi-request workflows easier and less error-prone thanks to the async context manager.
  • Keeps HTTP configuration and retry behaviour inside the hook.

LivyAsyncHook improvements

  • LivyAsyncHook now reuses the session and request logic provided by HttpAsyncHook session context manager.
  • Duplicate logic previously hidden in _do_api_call_async has been removed.
  • The hook no longer mutates internal state (e.g. self.method) when calling run_method.

Overall behaviour is unchanged, but the implementation is more DRY, predictable, and maintainable

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

dabla added 26 commits January 13, 2026 18:48
@dabla dabla changed the title Refactored HttpAsyncHook to easily support session based operations Refactor HttpAsyncHook to support session-based async HTTP operations and simplify LivyAsyncHook Feb 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant