[Bug] pydeck breaks streamlit with pandas 3

[lukasmasuch]

Description

I'm a maintainer of streamlit and we are working on supporting pandas 3. From our tests, it looks like there might be a compatibility issue with the json serializer:

.venv/lib/python3.13/site-packages/pydeck/bindings/json_tools.py:74: TypeError

def default_serialize(o, remap_function=lower_camel_case_keys):
    """Default method for rendering JSON from a dictionary"""
    if issubclass(type(o), PydeckType):
        return repr(o)
>   attrs = vars(o)
            ^^^^^^^
E   TypeError: vars() argument must have __dict__ attribute

The default_serialize function calls vars(o) on a weakref to a pandas DataFrame. In pandas 3.x, vars() fails because DataFrames no longer have a __dict__ attribute that vars() can access.

Flavors

• Script tag
• React
• Python/Jupyter notebook
• MapboxOverlay
• GoogleMapsOverlay
• CARTO
• ArcGIS

Expected Behavior

Works without an exception.

Steps to Reproduce

The error will be caused with Pandas 3, if a DataFrame is passed as data to a Layer, and .to_json() gets called on the pydeck object.

Environment

• Framework version: pydeck@0.9.1
• Browser: Chrome
• OS: MacOS

Logs

No response

[chrisgervang]

Thanks for the heads up. Any recommendations on how to fix this? Otherwise, I'll take a look at pandas upgrade guides and see what we can do.

Ideally, pydeck would be able to work with either pandas 2 or 3 since it's not relying on new functionality in pandas 3

[bodacious-bill]

Hello - I'm following this because I'm hoping for a quick update to streamlit for pandas 3.0 support...

It looks to me like this failure is related to this:

https://pandas.pydata.org/docs/whatsnew/v3.0.0.html#the-module-attribute-now-points-to-public-modules

From my limited skimming of the relevant pydeck code, it seems that if a pandas Dataframe is passed as data, it gets turned into a list of records:
pydeck/bindings/layer.py

...
        elif is_pandas_df(data_set):
            self._data = data_set.to_dict(orient="records")
...

But this update in pandas 3.0 breaks the is_pandas_df function to detect if the data passed is a pandas dataframe.

Here is a minimal example using your is_pandas_df function, reproduced below:

import pandas as pd

df = pd.DataFrame([],columns=['a','b','c'])
print(pd.__version__)

def is_pandas_df(obj):
    """Check if an object is a Pandas DataFrame

    Returns
    -------
    bool
        Returns True if object is a Pandas DataFrame and False otherwise
    """
    return obj.__class__.__module__ == "pandas.core.frame" and obj.to_records and obj.to_dict

if is_pandas_df(df):
    print(True)
else:
    print(False)
    print(df.__class__.__module__)

Output for pandas==2.3.3

2.3.3
True

Output for pandas==3.0.0

3.0.0
False
pandas

I'm not sure what is the best way to address this. Maybe something like this instead of obj.__class__.__module__ ?

(type(obj).__name__ == "DataFrame" and type(obj).__module__ == "pandas.core.frame") or (
    type(obj).__name__ == "DataFrame" and type(obj).__module__ == "pandas"
)

I hope this was helpful!

[lukasmasuch]

Yeah, the pandas.core.frame namespace/module is not correct anymore. We had to change the same aspect in Streamlit.

[chrisgervang]

Very helpful, thanks for detailing this.

So, .to_dict(...) is unchanged and it's just detection that broke?

@lukasmasuch does Streamlit have a detection function like this we can reference, and/or does @bodacious-bill'a proposed way of detecting seem like a good direction to go?

We should be able to release a quick patch of a small change like this - I'd just want to sanity check that there aren't any other regressions for pandas 3 users

[lukasmasuch]

Thanks for the quick fix!

So, .to_dict(...) is unchanged and it's just detection that broke?

yes 👍

@lukasmasuch does Streamlit have a detection function like this we can reference, and/or does @bodacious-bill'a proposed way of detecting seem like a good direction to go?

We are also just checking the full module path name, similar to how it's proposed above (e.g. here)