-
Notifications
You must be signed in to change notification settings - Fork 850
Open
Labels
Description
- Extend the
IngestionChunktype that can be found here with a public readonly property that returns the number of tokens used to represent given chunk:
public int TokenCount { get; }- Extend existing public constructor of this type with a mandatory
int tokenCountparameter. ThrowArgumentOutOfRangeExceptionwhen the value is negative. - Update all the chunkers that we ship in Microsoft.Extensions.DataIngestion project to provide this value at creation time.
- Update all the tests from the Microsoft.Extensions.DataIngestion.Tests project to provide this value as well.
- Add new tests or extend existing ones with explicit token count verification.
- Ensure there are no build errors.
This is the first step toward implementing #6971
Reactions are currently unavailable