Implement auto_new attribute for #[pyclass]

[RedKinda]

PR for issue #5416

This introduces a new #[pyclass] attribute called auto_new, where set_all is required to be used alongside it. It automatically generates a #[new] function for this class, similar to that of the python dataclass. It requires the multiple-pymethods feautre, as it generates its own #[pymethods] block

[RedKinda]

I am not quite sure why the CI failing, I dont see any failure in the logs

[davidhewitt]

Thanks for the PR, and sorry for the very slow review.

I think this feature makes a ton of sense, I've wanted something like this for ages. auto_new is maybe the right name, the only alternative I know is something like dataclass where this is roughly equivalent to the dataclass(init=True) option. I don't think init is correct here though, since this generates __new__.

To resolve before merge: #5284 proposes #[pyclass(default)] which would allow a constructor which takes no arguments and uses Default::default(). It might be nice to not have separate default and auto_new attributes. What if we had #[pyclass(auto_new = "from_fields")] (this PR) and #[pyclass(auto_new = "default")] ? Name and design ideas welcome.

Other open questions, potentially fine to leave for follow ups:

• We could allow #[pyo3(default = X)] annotations on the fields to construct them.
• We could consider #[pyo3(kw_only)] at both the class and field level.

The CI failure is UI tests, you can fix with nox -s update-ui-tests:

https://github.com/PyO3/pyo3/actions/runs/17557400301/job/49865008001?pr=5421#step:19:3264

[davidhewitt]

We don't need this restriction (at the cost of a slightly more complicated implementation) - see generate_protocol_slot and how it's used for __str__.

[RedKinda]

Hi, thanks for the review!

In regards to the defaults-related PR, #[pyclass(auto_new = "from_fields")] and #[pyclass(auto_new = "default")] make sense to me, names sounds fine.

As for the followups, I would leave those to be in a separate PR and get this and the default one out first. I don't have the capacity to work on them at the moment but maybe if we open an issue someone will pick it up. I am also gonna look at the generate_protocol_slot comment before merging, but might take a bit since I am not familiar with it yet.

[Icxolu]

auto_new is maybe the right name, the only alternative I know is something like dataclass

Perhaps we could just call it new. The "auto" feels a bit redundant to me. Just setting the option already implies that something is happening. We don't call it auto_str for example, just str.

[RedKinda]

@davidhewitt in regards to generate_protocol_slot for auto_new should that be a new pub const __NEW__: SlotDef? not sure what else but just confirming

@Icxolu i think it might be a little confusing for newbies reading docs to have both #[pyclass(new)] and #[new] on functions, but i don't feel strongly about this

[davidhewitt]

Good question, #5551 produces a __NEW__ slotdef which you should be able to use, I'm going to split that diff up. Feel free to either base off that PR or give me some time to merge it in stages (might take a few days).

I think #[pyclass(new)] seems correct to me, this matches #[pyclass(str)] and __str__. I would quite like to have #[new] optionally just be __new__.

[davidhewitt]

With #5551 merged, this should be able to rebase on top now.

[RedKinda]

I am running into an issue, I think generate_protocol_slot assumes first argument is self, and generates broken code when it isn't. Image below shows cargo expanded version of this simple test struct.

#[pyclass(get_all, set_all, new = "from_fields")]
struct AutoNewCls {
    a: i32,
    b: String,
    c: Option<f64>,
}

I am not sure how to go about fixing this properly @davidhewitt maybe you could take a look? I pushed the latest broken commit with it, other than that it should work but hard to tell with this being broken.

[davidhewitt]

Thanks for following up on this, and sorry for the delay from my end. I pushed a fix which has CI passing now.

Before merging, I have a few final points of discussion and test suggestions. See comments below.

[davidhewitt]

It would be good to add a test with a tuple struct, e.g.

struct Point2d(f64, f64);

... I would think the generated constructor would allow only positional inputs, as there are no names for the fields.

[davidhewitt]

(It might be good enough for this to gracefully fail with a "not yet supported" message as far as this PR is concerned.)

[RedKinda]

unless i am missing something i added support for tuple structs here 4881636

[davidhewitt]

We should have a quick think about the interaction of #[pyo3(new = "default")] and the other future extensions I suggested in #5421 (review)

• Should new = "default" accept any arguments? The easy answer is no. But what if users want to have a constructor which is the equivalent of MyStruct { x, y, ...Default::default() }, i.e. use the struct-level default except for some specific fields?
• Should fields accept #[pyo3(new = <value>)] to remove them as arguments from the generated constructor and always set them to <value> (similar to dataclasses.field(init = False))? This would appear to have very strong overlap with #[pyo3(new = "default")]on the struct, presumablyfor the field would come from theDefault` implementation.
• Once we have both of these, what is the difference between "from_fields" with a bunch of #[pyo3(default = <value>)] annotations on fields?

... it feels to me like the general design would be that new = "from_fields" would not require a Default implementation, and would allow users to take fields out of the constructor and instead give them default values via #[pyo3(default = <value>)].

new = "default" would be the opposite; it would require a Default implementation and would require users to opt-in to add fields to the constructor to allow callers to override the value set by the Default implementation. It feels like #[pyo3(new = true)] might be the right annotation for this, but I can't decide. That can in theory be a future extension for new = "default" so maybe it's a long way off.

We don't need to solve this now, but I'd at least like to make sure that having "from_fields" as implemented here doesn't accidentally close off the design space. We can write this all into a follow-up issue after merge.

[RedKinda]

If using new = "from_fields", i think field-level defaults should make that field a keyword argument in __new__ and give it the configured default. Example:

#[pyclass(new = "from_fields")]
struct Foo {
    a: u64
    #[pyo3(default=5)]
    b: u64
}

would be the equivalent of

class Foo:
    def __new__(a: int, b: int = 5) -> Self:
        ...

Using new = "default" could then simply always produce an empty __new__.

[davidhewitt]

What if a user didn't want b to be user-providable at all? Maybe a #[pyo3(new_arg = false)] on the field e.g. maybe this:

#[pyclass(new = "from_fields")]
struct Foo {
    a: u64
    #[pyo3(new_arg = false, default=5)]
    b: u64
}

would produce Python API

class Foo:
    def __new__(a: int) -> Self:
        ...

and the Rust implementation would be equivalent to

#[pymethods]
impl Foo {
    #[new]
    fn new(a: u64) -> Self {
        Self {
            a,
            b: 5  // if `new_arg = false` was not set, then `b` would still be set here but just with default of 5
        }
    }

---

Using new = "default" could then simply always produce an empty __new__.

Seems reasonable, and users could add arguments with #[pyo3(new_arg = true)]? e.g.

#[pyclass(new = "default")]
struct Foo {
    a: u64
    #[pyo3(new_arg = true, default=5)]
    b: u64
}

would produce Python API

class Foo:
    def __new__(a: int, b: int=5) -> Self:
        ...

and a Rust implemementation equivalent to

#[pymethods]
impl Foo {
    #[new]
    #[pyo3(signature = (b = 5))]
    fn new(b: u64) -> Self {
        Self {
            b,
            ..Self::default()
        }
    }

[RedKinda]

looks like nox -s update-ui-tests isnt working for me :/ I think I am running on a different rust version producing slightly different errors, as it slightly modifies unrelated tests as well

[davidhewitt]

I pushed a commit which should fix. The &str extraction only works like that on Python 3.9 and older, perhaps caused by your system Python? I am unsure why we don't notice that as a problem in our CI 🤔

We use latest stable Rust, also latest trybuild version (rustup update stable && cargo update should align you with CI).

[davidhewitt]

I will give another pass at review probably on Sunday.

[davidhewitt]

Looks like the &str case in the UI test is indeed still a problem. I'm currently investigating that code to see what we can do to improve the error message.

[davidhewitt]

Circling back here, I have some ideas how to solve that problem, but the async code being heavily special cased (xref #5681) is causing issues for me. If I get that code adjusted to not have the special case I think I can land cleanups to unblock here.

[codspeed-hq]

Merging this PR will not alter performance

✅ 100 untouched benchmarks

---

_{Comparing RedKinda:dev/auto_new (3c9a12a) with main (d7c050c)}

[davidhewitt]

I am really struggling to reproduce the UI test failure locally, very confusing 😖

[davidhewitt]

I have narrowed the failure to dtolnay/trybuild#324

[davidhewitt]

Looks like this is probably ready to merge. Anyone else want to take a read of this before I click the button? Thanks again @RedKinda @Icxolu

[Icxolu]

Maybe we should have a test that using the new option together with a manual #[new] constructor is a compile error?

[davidhewitt]

Good idea, pushed

[Icxolu]

Great, thanks for pushing this through!