Any faster approach to read data values from channels

[YashKarpe96]

currently i am using an approach like below

auto *dg = dataGroups[dgIndex];
auto *cg = dg->ChannelGroups()[cgIndex];

std::vector<double> yData;
double yEngValue = 0.0;

mdf::IChannel *channel = cg->GetChannel(channelName);

std::unique_ptr<mdf::IChannelObserver> ySub = CreateChannelObserver(*dg, *cg, *channel);

reader->ReadData(*dg);
mdf::IChannelObserver &yObs = *ySub;

for (size_t sample = 0; sample < yObs.NofSamples(); ++sample) {

    if (yObs.GetChannelValue(sample, yEngValue)) {
        yData.append(yEngValue);

    }

}

but doing this everytime retrieving a channel data is too time consuming. i am not able to work around a way to improve the speed to retrive the data.

[ihedvall]

@YashKarpe96
If you add some timing in your code, you hopefully will notice that it is the reader->ReadData(*dg) line that consumes most (~95%) of the time. If not there is some optimization you can do as getting all values in one call and so on. Try to add some timers in your code

The MDF store the samples record by record. A record size is define by its channel group (CG). This is good for the logger/writers as they just append the records to the end of the file. The penalty comes when reading a specific channel data. This is common problem with MDF files so there is some solutions. Below I assume that you wants all sample values for a specific channel. If you want partial samples read, there are some optimize methods.

1. If you are lucky, the MDF file is sorted. It is possible to optimize the read but this doesn't speed up the read so much.
2. The other approach is to read in all sample once into the observers. Reading one channel or read all channels take the "same" time. Well you need to keep everything in primary memory.
3. Same idea as solution 2 but convert to a file format that is faster to read. HDF5 and MATLAB files is typical examples.
4. Using solution 3 but adding an index database that keeps track of all values and where they are store. This solution is best if you want to keep track of many files.

I don't know if this answer is what you expected but there is no single solution for this. What is your requirement? You may set up a MS Teams (or similar) meeting as there is a lot of solutions involved.

Best Regards
Ingemar Hedvall

[YashKarpe96]

Hello Mr Hedvall,

Thanks a lot for your detailed response. I really appreciate the insights you've provided. I was actually already in the process of reading the channel data at the start itself and storing it in memory, but it was still consuming more time than expected, so I just wanted to explore if there was a better approach before finalizing an approach.

For now, I'll try out the method 3 as well which you suggested.

I really appreciate the offer for a meeting, but as I'm still quite new to C++, I believe my code is still quite raw and needs significant refinement. I think it would be best for me to work on improving it on my own first, to avoid taking up too much of anyone's time.

Thank you again for your support.

Best regards,
Yash

[ihedvall]

@YashKarpe96
Yes, C++ is sometimes tricky. Not the C++ code but compiling and linking can be a challenge. The linker error messages often makes no sense.

I just did an MDF to CSV converter (mdf2csv.exe). Let me know if you need an MDF to HDF5 converter. That converter can be general and open source.

Just one simple improvement that I forget earlier. Reading from a network disc (NAS) or a local SSD disc are big speed differences. So copy the file first to SSD and then read the file might give some better speed.