BUG: Pandas converts nullable int to float, even when this loses data by kjmin622 · Pull Request #63925 · pandas-dev/pandas

kjmin622 · 2026-01-29T04:52:52Z

closes BUG: Pandas converts nullable int to float, even when this loses data #63903 (Replace xxxx with the GitHub issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.
I have reviewed and followed all the contribution guidelines
If I used AI to develop this pull request, I prompted it to follow AGENTS.md.

Summary

Fix precision loss when using Series.apply() or Series.map() on nullable integer dtypes (Int64, UInt64, etc.) with None values.

Problem

When applying a function to a Series with nullable integer dtype containing NA values, the data was being converted to float64, causing precision loss for large integers that exceed float64's integer precision limit (2^53 ≈ 9×10^15).

import pandas as pd
def add_two(x):
    if pd.isna(x): 
        return pd.NA 
    return x + 2
sequence = [10000000000000001, None] # above float64 precision limit
ser = pd.Series(sequence, dtype='Int64')
result = ser.apply(add_two)

Before: 10000000000000002 (wrong - precision lost)
After: 10000000000000003 (correct)

Solution

Modified BaseMaskedArray.map() to:

Use to_numpy(dtype=object, na_value=pd.NA) instead of to_numpy() to preserve integer values
Apply _cast_pointwise_result() to restore the appropriate nullable dtype

kjmin622 · 2026-01-30T06:39:31Z

@aaron-seq Thank you for the review.

As you suggested, adding a preserve_dtype parameter would eliminate the breaking change and remove potential issues. I will implement it.

However, all tests are currently passing. Only one test failed, but it is caused by #63936 and is unrelated to the current code changes.

mroeschke · 2026-01-30T19:22:50Z

@aaron-seq do not post AI generated pull request reviews again. Please review our AI policy. Similar contributions in the future may lead to a ban.

aaron-seq · 2026-01-31T04:56:35Z

@aaron-seq do not post AI generated pull request reviews again. Please review our AI policy. Similar contributions in the future may lead to a ban.

Thanks for this, will note this when contributing in future

kjmin622 · 2026-01-31T15:06:22Z

As you suggested, adding a preserve_dtype parameter would eliminate the breaking change and remove potential issues. I will implement it.

Instead of adding preserve_dtype, the map function was modified to return the same dtype as before.

rhshadrach

Thanks for the PR!

pandas/tests/series/methods/test_map.py

pandas/tests/apply/test_series_apply.py

kjmin622 · 2026-02-03T01:55:47Z

pandas/core/arrays/masked.py

+            try:
+                return type(self)._from_sequence(result, dtype=self.dtype)
+            except (ValueError, TypeError):
+                return result


This code will preserve the type if it can be preserved.

Agreed we should return a masked array here, but if the user returns floats we should not convert them back to integers even if they have no fractional value. E.g.

ser = Series([1, 2, 3], dtype="Int64") result = ser.apply(lambda x: 3.0)

should result in Float64. Just use self._from_sequence I think.

This can use self._cast_pointwise_result instead of _from_sequence, I think. That latter function was exactly added to be used in those kind of situations

xref #62164

kjmin622 · 2026-02-03T04:05:47Z

Thanks for the PR!

@rhshadrach Thank you for your review. I reflected them.

rhshadrach

I'm positive on preserving masked EAs in map/apply, things like this would be useful especially when making NumPy-nullable the default. But this can be a large break for current users. I'm thinking this could need a deprecation instead. Perhaps if we are going to make a feature flag for NumPy-nullable as a default this would go behind it?

cc @jbrockmendel @jorisvandenbossche @mroeschke

rhshadrach · 2026-02-05T10:52:59Z

pandas/core/arrays/masked.py

+            try:
+                return type(self)._from_sequence(result, dtype=self.dtype)
+            except (ValueError, TypeError):
+                return result


Agreed we should return a masked array here, but if the user returns floats we should not convert them back to integers even if they have no fractional value. E.g.

ser = Series([1, 2, 3], dtype="Int64") result = ser.apply(lambda x: 3.0)

should result in Float64. Just use self._from_sequence I think.

rhshadrach · 2026-02-05T10:55:03Z

doc/source/whatsnew/v3.0.1.rst

 - Fixed a bug in :func:`col` where unary operators (``-``, ``+``, ``abs``) were not supported (:issue:`63939`)
 - Fixed a bug in the :func:`comparison_op` raising a ``TypeError`` for zerodim
  subclasses of ``np.ndarray`` (:issue:`63205`)
+- Fixed bug in :meth:`Series.apply` and :meth:`Series.map` where nullable integer dtypes were converted to float, causing precision loss for large integers (:issue:`63903`)


I believe this is not a regression, note should be in 3.1.0. Also need to note map/apply are now preserving NumPy-nullable EAs.

jbrockmendel · 2026-02-05T17:02:43Z

I'm positive on preserving masked EAs in map/apply, things like this would be useful especially when making NumPy-nullable the default. But this can be a large break for current users. I'm thinking this could need a deprecation instead. Perhaps if we are going to make a feature flag for NumPy-nullable as a default this would go behind it?

I lean towards "treat this as a bugfix" since "preserve dtype backend" is a mostly-consistent policy. But not a super-strong opinion.

rhshadrach · 2026-02-10T22:16:04Z

doc/source/whatsnew/v3.1.0.rst


 Other enhancements
 ^^^^^^^^^^^^^^^^^^
+- :meth:`Series.apply` and :meth:`Series.map` now preserve nullable (masked) extension array dtypes where appropriate; e.g. when the result is float, the output dtype is ``Float64`` rather than being cast back to the input dtype (:issue:`63903`).


I think this can be removed and...

rhshadrach · 2026-02-10T22:16:28Z

doc/source/whatsnew/v3.1.0.rst


 ExtensionArray
 ^^^^^^^^^^^^^^
+- Fixed bug in :meth:`Series.apply` and :meth:`Series.map` where nullable integer dtypes were converted to float, causing precision loss for large integers (:issue:`63903`).


...add a little detail here.

Suggested change

- Fixed bug in :meth:`Series.apply` and :meth:`Series.map` where nullable integer dtypes were converted to float, causing precision loss for large integers (:issue:`63903`).

- Fixed bug in :meth:`Series.apply` and :meth:`Series.map` where nullable integer dtypes were converted to float, causing precision loss for large integers; now the nullable dtype will be preserved (:issue:`63903`).

rhshadrach · 2026-02-10T22:17:18Z

pandas/core/arrays/masked.py

+            mapper,
+            na_action=na_action,
+        )
+        if isinstance(result, np.ndarray):


I think this should always be a NumPy array; can you check? Change this to assert isinstance(result, np.ndarray) and see that tests still pass.

@rhshadrach I confirmed that the test passes even when I add assert isinstance(result, np.ndarray). Thank you.

rhshadrach

lgtm

rhshadrach · 2026-02-11T21:23:20Z

@jbrockmendel @mroeschke - plan to merge in a few days if you want to take a look.

kjmin622 and others added 4 commits January 29, 2026 13:47

BUG: Pandas converts nullable int to float, even when this loses data

9c7be95

modify test

39cd95a

modify test_masked

f025c9f

Merge branch 'main' into MapReturnValue

ea1bac2

This comment was marked as spam.

Sign in to view

Return value same as before (PR#63925)

5ebba13

kjmin622 force-pushed the MapReturnValue branch 2 times, most recently from f29bebb to 5ebba13 Compare January 31, 2026 15:01

kjmin622 and others added 2 commits February 1, 2026 00:02

modify test (PR#63925)

48c87b3

Merge branch 'main' into MapReturnValue

186479b

kjmin622 and others added 2 commits February 1, 2026 00:15

pre-commit (PR#63925)

f508f08

Merge branch 'main' into MapReturnValue

9730c29

rhshadrach requested changes Feb 2, 2026

View reviewed changes

pandas/tests/series/methods/test_map.py Outdated Show resolved Hide resolved

pandas/tests/series/methods/test_map.py Outdated Show resolved Hide resolved

pandas/tests/series/methods/test_map.py Show resolved Hide resolved

pandas/tests/apply/test_series_apply.py Show resolved Hide resolved

apply review (PR#63925)

4f98070

kjmin622 commented Feb 3, 2026

View reviewed changes

kjmin622 and others added 2 commits February 3, 2026 10:58

Merge branch 'main' into MapReturnValue

64c9aa1

pre-commit (PR#63925)

46ad0c4

Merge branch 'main' into MapReturnValue

8da87ab

rhshadrach requested changes Feb 5, 2026

View reviewed changes

apply _cast_pointwise_result (PR#63925)

05dcc23

rhshadrach requested changes Feb 10, 2026

View reviewed changes

reflect review (GH#63925)

763cb2d

rhshadrach added Bug NA - MaskedArrays Related to pd.NA and nullable extension arrays labels Feb 11, 2026

rhshadrach added the Apply Apply, Aggregate, Transform, Map label Feb 11, 2026

rhshadrach approved these changes Feb 11, 2026

View reviewed changes

rhshadrach added this to the 3.1 milestone Feb 11, 2026

	- Fixed bug in :meth:`Series.apply` and :meth:`Series.map` where nullable integer dtypes were converted to float, causing precision loss for large integers (:issue:`63903`).
	- Fixed bug in :meth:`Series.apply` and :meth:`Series.map` where nullable integer dtypes were converted to float, causing precision loss for large integers; now the nullable dtype will be preserved (:issue:`63903`).

Uh oh!

Conversation

kjmin622 commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Solution

Uh oh!

This comment was marked as spam.

kjmin622 commented Jan 30, 2026

Uh oh!

mroeschke commented Jan 30, 2026

Uh oh!

aaron-seq commented Jan 31, 2026

Uh oh!

kjmin622 commented Jan 31, 2026

Uh oh!

rhshadrach left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jorisvandenbossche Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kjmin622 commented Feb 3, 2026

Uh oh!

rhshadrach left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jbrockmendel commented Feb 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rhshadrach left a comment

Choose a reason for hiding this comment

Uh oh!

rhshadrach commented Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

kjmin622 commented Jan 29, 2026 •

edited

Loading

jorisvandenbossche Feb 5, 2026 •

edited

Loading

rhshadrach left a comment •

edited

Loading