Skip to content

[FEA] Batching embedding copy in Dynamicemb #302

@shijieliu

Description

@shijieliu

Is your feature request related to a problem? Please describe.
Right now, in dynamicemb, for embeddings, there are multiple times memory copy happening in the forward and backward, which hurts the performance. To achieve best practice, we should batch those copies.

Describe the solution you'd like

  1. For HBM mode, remove intermedia unique_emb buffer
  2. For cache mode, reuse forward results in backward to avoid touching non-HBM storage multiple times in single iteration
  3. Fuse insert with update for training.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context, code examples, or references to existing implementations about the feature request here.


By submitting this issue, you agree to follow our code of conduct and our contributing guidelines.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementImprovement for existing feature

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions