NB#16 LDA Topics with impresso-pipelines Package

[caiocmello]

Notebook file: LDA Topics with impresso-pipelines Package

Comments and feedback on this issue are welcome

[caiocmello]

Dear @maslionok. Thanks once again for a great NB! The structure of the NB looks good and the text reads well.

I will add here some initial feedback I believe might be worth addressing:

1. As far as I understand, LDA topics have been already calculated for the Impresso Corpus, and we have a list of topics in the model. What the model does for external texts is assigning them to one of those topics (not calculating new topics). Is this correct? If so, it might be worth making this clearer. It is important that people understand we are not calculating new topics, as done when using traditional topic modelling but assigning new input text to existing pre-calculated topics.
2. For some reason, the code worked for me on Google colab but not locally using VS Code. When I run from impresso_pipelines.ldatopics import LDATopicsPipeline, I get the error: ModuleNotFoundError: No module named 'impresso_pipelines.ldatopics'.
3. On the 'interpretation of the result' section: do you think we could add a piece of code to open the json file and print the keywords/label of the most relevant topic in the notebook? This might facilitate the process for those unfamiliar with Json files.
4. Should we add a section on explainability/interpretability of LDA topics? I wonder if you have any suggestions on how to address this issue. Can we find clues on why the model has assigned the input text to that specific topic?

That's all for now. Let me know if you have any questions/comments.
and thanks again!

[maslionok]

Dear @caiocmello, thank you so much for the review, it really helps!

I have went through each of your comments and here is the summary:

1. I have added a small comment "earlier pre-computed topic" to the "How it works" section. Do you think this is enough or should we clarify it in other parts of the notebook?
2. This is very strange, I have double checked and for me it works. From what I know if you are using mac machine, to install the package you must type pip install "impresso_pipelines[ldatopics]" in the terminal (use ""). Could you please elaborate a bit on the problem? Was the package installed correctly?
3. Do you mean to take from the output of ldatopics_pipeline(text, diagnostics_topics=True) only the diagnostics_topics field and print it separately?
4. Hm, overall I see the idea of that and I think it is important to make it clear! Currently the explainability we have comes from diagnostics_topics and diagnostics_lemmatization. User can see what words from the original text are passed into topics assigning model and then understand a bit each assigned topic using top 10 words from each topic. What we could do is put this into a separate section, something like "Explainability", but then we will slightly change the whole format of the notebook, which I am not sure is a good idea. Let me know what you think about doing so!

Once again thank you!

[caiocmello]

Dear @maslionok,

I went through the notebook once again. Thanks very much for addressing the previous comments. Everything seems to work smoothly now. I've added a couple comments on the pull request (mostly typos). Thanks once again. As soon as you finish fixing those, you can merge into main and we will assign a reviewer to this notebook.

[maslionok]

Dear @caiocmello, thank you for going through it again, I really appreciate it. I have finished the edits that you have pointed out and completed the pull request.

[caiocmello]

This notebook will be reviewed in Phase 2 by @Ferdaous-af. Please, refer to the reviewer's guidelines below and add your feedback as a comment on this issue. Any further comments are also welcome!

Deadline for this task is the 4th of July 2025. Please, let me know in case you need more time to complete your review.

Notebook file: LDA Topics with impresso-pipelines Package

Internal review:

• Is the code consistent? Eg. Use of variables, formatting
• Is the explanation of the code correct? Is there imprecise information? Could we expand it in some aspect you consider important to understand the code?
• Are there any references to external resources that could enrich this notebook?
• Is the information (text in markdown) contained in the NB enough to perform the proposed task? Is there something missing?
• Do the objectives under 'what you will learn' match the content? Do we provide everything we promise in the objectives?

[VeprikovK]

Good afternoon, @caiocmello!

There is my short review on the notebook LDA Topics with impresso-pipelines Package.

General impression.
In general it's quite clear code notebook, with understandable main idea - to show how you can analyse key topics in the text.

General comment:
While it quite clear how to analyse the separate individual textes, it's not clear how it can be implemeted to the massive of textes from IMPRESSO. For example to analyse the fe hundreds of text to select ones with the same prevalent topic. However imaybe it topic for the other notebook.

Specific comments:
Narrative aspect

doc_name: A string identifier with no spaces (e.g., "my_document"). When you provide this, it replaces the autogenerated document ID. The internal counter only increments for calls where you don’t specify a name, so the next unnamed document will pick up from the following number.

This section feels a bit confusing, I had to read it few times to understand correctly. The matter is that for another attributes the description wrote in the scheme: (attribute name) + verb. Thus it explain wht you can do with this attribute. For the doc_name there is no verb, thus initially I decided that it not function to create unique name, but the function to show you this "string identifier". Probably it would be better to explain it in the same scheme that other attribute, like:

doc_name: allows you to create a unique name for the text instead of autogenerated document ID. (then explain the requirements fot he name, like it should be without space etc.

Code aspect
When I run the next sequence:

from impresso_pipelines.ldatopics import LDATopicsPipeline
pipeline = LDATopicsPipeline()

It shows me the next error notification:

RuntimeError: Could not find JVM. Please install Java and/or set JAVA_HOME environment variable. Error: No JVM shared library file (libjvm.so) found. Try setting up the JAVA_HOME environment variable properly.

It didn't allow me to launch the analyse until I install the packages.