Conversation
josevalim
reviewed
May 12, 2024
| debug? = Keyword.get(jit_opts, :debug, false) | ||
|
|
||
| sample_data = | ||
| case Enum.take(data, 1) do |
Contributor
There was a problem hiding this comment.
This is tricky...
If the stream is pointing to an external resource, then you are opening it and traversing it. If the stream has side-effects, this first element is effectively lost. The correct way is to do this when streaming starts. Something like this:
Stream.unfold({:init, arg1, arg2}, fn
{:init, arg1, arg2}, data ->
state = init_loop_state(...)
{emit_next(state), {:state, state}
{:state, state}, data ->
....
:halt, _data ->
...
end)
Contributor
There was a problem hiding this comment.
I think you would have to move everything to init, including fire_event(:started, ...).
josevalim
reviewed
May 12, 2024
| loop | ||
| |> stream(data, init_state, opts) | ||
| |> Stream.take(max_epochs) | ||
| |> Stream.with_index() |
Contributor
There was a problem hiding this comment.
Do you need with_index if you already have state.epoch?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This takes the regular loop
runand builds it based off of thisstreamwhich will return a loop state per epoch. Basically this should be really flexible in that I can now control the running of the loop with regular ElixirStreamandEnumfunctions. You can see this in how much simpler the olderrunis now.This also kills
output_transformin the loop. I may add it to theLoop.runas an option and that case we can add some output transforms that will not break some older training codeResolves #362
Will potentially resolve #455 depending on how my experiments go