Packages best practices

[labdmitriy]

Hi @rasbt,

I remember we discussed multiple things about Python packages, yesterday I finished completely restructured article about packages and distributions, and I mentioned 4 cookiecutter templates there from very experienced people, along with online book and series of articles - https://labdmitriy.github.io/blog/distributions-vs-packages/#information-from-the-internet

I think that it could be useful for you to this and new packages that you create.

[rasbt]

Wow nice! Downloaded and put it on my e-reader for my upcoming trip!

[labdmitriy]

Great, it would be interesting to hear feedback about the article and provided links whether it is useful for you.

[rasbt]

Just read through it! Very interesting analysis. I was very confused about the naming conventions as well. First, library, package, module etc. is something I tried to find out myself a few years ago. I personally came to the conclusion that there is a lot of ambiguity around it, and I just forgot to worry about it 😅.

The other thing, regarding hyphens: Originally, I preferred underscores for file names and also packages. Esp. for Python packages it allows you to have the distribution name match the import name -- scikit-learn vs sklearn is always something I found annoying about scikit-learn (and there are many other examples).

On the other hand, for file names, I heard that on Google, hyphens are preferred. Just looked it up and found this when I searched for it ... on Google:

Google recommends using hyphens instead of underscores in your URLs. This is because using hyphens instead of underscores makes it easier for Google's web crawler to compute the information that your website has and create consistent results.

Afaik the same thing goes for Finder on macOS. Say I named a file but forgot its exact name but can only remember a few words. I noticed that typing these words often helps me find hyphenated files more easily. I think with underscores you often need a precise match.

[rasbt]

That being said, I just remember that I was thinking about this very hard when I created https://github.com/Raschka-research-group/coral-pytorch. The distribution name is hyphenated, the package name is coral_pytorch though. I thought it might be fine since others are doing it this way as well (pytorch-lightning, scikit-learn). But yeah, somehow it also doesn't feel right and I always search for my coral_pytorch repo 😅

[labdmitriy]

Hi Sebastian,

Just read through it! Very interesting analysis. I was very confused about the naming conventions as well. First, library, package, module etc. is something I tried to find out myself a few years ago. I personally came to the conclusion that there is a lot of ambiguity around it, and I just forgot to worry about it sweat_smile.

It is great that it was useful for you!
Did you follow the links to cookiecutter templates and related online book ans article series? I think that it is much more valuable than my article :) I saw that you are creating PyTorch Lightning engineering class and it can be especially useful for you now.

The other thing, regarding hyphens: Originally, I preferred underscores for file names and also packages. Esp. for Python packages it allows you to have the distribution name match the import name -- scikit-learn vs sklearn is always something I found annoying about scikit-learn (and there are many other examples).

Moreover if you try to install scikit-.-__-.-learn you will also install scikit-learn successfully as mentioned here :)

On the other hand, for file names, I heard that on Google, hyphens are preferred. Just looked it up and found this when I searched for it ... on Google:
Google recommends using hyphens instead of underscores in your URLs. This is because using hyphens instead of underscores makes it easier for Google's web crawler to compute the information that your website has and create consistent results.

Yes, it probably corresponds to the answer of Tomas Beuzen.

Afaik the same thing goes for Finder on macOS. Say I named a file but forgot its exact name but can only remember a few words. I noticed that typing these words often helps me find hyphenated files more easily. I think with underscores you often need a precise match.

That being said, I just remember that I was thinking about this very hard when I created https://github.com/Raschka-research-group/coral-pytorch. The distribution name is hyphenated, the package name is coral_pytorch though. I thought it might be fine since others are doing it this way as well (pytorch-lightning, scikit-learn). But yeah, somehow it also doesn't feel right and I always search for my coral_pytorch repo 😅

Interesting notes, thank you!

[rasbt]

Moreover if you try to install scikit-.-__-.-learn you will also install scikit-learn successfully as mentioned here :)

Haha, fair enough. However, I think it only works in pip but not in conda. In pip you can also do pip install sklearn apparently. Didn't know. That's neat.

Did you follow the links to cookiecutter templates and related online book ans article series?

Unfortunately, I haven't had the chance, yet. I have been using a customized version data science cookiecutter for some internal projects though (https://github.com/Raschka-research-group/newproject-github). Hope to get a chance to check out these other resources as well some time.

[labdmitriy]

Haha, fair enough. However, I think it only works in pip but not in conda. In pip you can also do pip install sklearn apparently. Didn't know. That's neat.

As I remember, sklearn in PyPI is not the same as scikit-learn and version 0.0 will be installed without any functionality :) - https://pypi.org/project/sklearn/

Unfortunately, I haven't had the chance, yet. I have been using a customized version data science cookiecutter for some internal projects though (https://github.com/Raschka-research-group/newproject-github). Hope to get a chance to check out these other resources as well some time.

Great! I started also with cookiecutter-ds (version 1) and realize that it has some flaws for my tasks and from the Python package perspective, so I am changing it gradually.

And I also switched to black formatter after contributing to your library :)
By the way, I prepared webinar for our team about contribution to mlxtend and detailed explanation about contributor-mantainer interaction and will use this material for the article.

[rasbt]

Yeah, that's what I thought, but for some reason it then got the cached version of the recent sklearn I had on my computer.

Btw. regarding the formatting, that was a very useful convo! Just put together an engineering class (not published yet) on using isort -> black -> flake8 yesterday :). Glad to hear the convo was useful to you as well!

[labdmitriy]

Yeah, that's what I thought, but for some reason it then got the cached version of the recent sklearn I had on my computer.

Oh I didn’t notice that about sklearn, it’s like an old stub, great)

Btw. regarding the formatting, that was a very useful convo! Just put together an engineering class (not published yet) on using isort -> black -> flake8 yesterday :). Glad to hear the convo was useful to you as well!

Of course it was extremely useful!