Training Questions

[jhunter533]

I wanted to note I appreciate the from_torch tutorials that were recently uploaded and the help they provide. I wanted to ask some clarifying questions.
When using the from_torch function does the original network need to actually use sequential or will it see through forward calls in something like

class Network(nn.Module):
  def __init__():
     self.state_fc=nn.Linear(num_states,256)
  def forward(self):
     state_out=F.relu(self.state_fc(state))
     return torch.squeeze(layer)
  def train():
     #Training lines

When doing in cross-sim training to fine tune a network converted using from_torch I understand you adding synchronize() but if in training I originally called self.Network do I keep that the same only adding synchronize or do I change it to a call for the converted network and add synchronize?
Additionally, if I was running training with multiple networks and soft updates do all networks need to be converted or just the main networks (not target networks using soft updates)?
Finally, I wanted to ask if there was support for lstm layers or plans to add them.

Thank you for any help it's very appreciated.

[bfeinberg]

1. from_torch() uses the Module.named_children() function to enumerate potential layers to convert, so in the example provided above it would convert state_fc into an AnalogLinear layer. If you want to know what layers CrossSim will convert within a network you can use use the convertible_modules() function from the same simulator/algorithms/dnn/torch/convert.py file.
2. I believe in this case you would be able to call train on the converted Network object but you would need to add synchronize after each call to the optimizer so the in-place weight updates are reflected in the optimizer.
3. I'm not sure what you mean by soft updates, can you explain more. If you are asking about recursive combinations of networks then a single convert call should convert everything, it recursively traverses the model.
4. Yes, LSTM and other forms of recurrent layers are something we have on our todo list although I'm not entirely sure when we'll get around to them. If this is a priority for your work we can give you some suggestions on how to implement it.

[jhunter533]

Sorry I realize some of my questions were a bit vague.

1. I observed it said the layers when using convertible_modules() for my example but it did not say the activation call like it does when checking a model given in sequential. So I couldn't tell if the function was properly seeing the activation calls since they are done layer by layer in forward.
2. The specific code I was using was a class containing a training function and it called another class object of the network. I noticed that when I was testing the conversion on just a test outside of classes I had to call the converted network not the original when testing. So I was unsure if every call in training that I started to retune the network weights should be to the original network with synch added or the converted one. Honestly, this might be a problem with my structure.
3. I'm specifically trying to implement something more related to reinforcement learning so I was testing things with ddpg which utilizes soft updates of the target networks to then update the real network so I wasn't sure if in retuning in cross-sim if I should convert target networks to update the original or if like in the previous just synch fixes this. I probably just need to experiment with it more.
4. The plan is to eventually use LSTM layers but they are not a top priority with what I'm doing so I may eventually follow up on that if they are not added by then.
   I appreciate the help again.

[gwchapm]

1: If you just print the network post conversion (print(from_torch(my_network, configs)) if should list some layers being replaced with AnalogLinear or AnalogConvolutional.

2 & 3: You should be calling synchronize wherever you want to perform an update on the analog network. If you have multiple agents within a ddpg "inner-loop" that somehow create a single update step then you could call only on the outer loop. Otherwise, you'll need to call for each instantiation of a network on each optimizer step.

4: You can see an example of how we implemented a simple RNN in the NICE tutorial, notebook 3, essentially re-implementing torch.nn.RNNCell with linear/convolutional layers (which can be converted). You could do something similar for LSTMCell. This won't do an analog simulation of the elementwise operations, but will atleast implement the various weight matrices.