How to convert WDC dataset to your JSON format?

[lefterisloukas]

I've downloaded the dataset but it needs some pre-processing to get it to your JSON format, as in the sample you provide in the repo.
Do you have the scripts for this process?

[HaoAreYuDong]

We will sort out the code and make it public soon. Please stay tuned!

[lefterisloukas]

Thanks for your reply @HaoAreYuDong.

Until you sort the code and publish it, could you maybe please share the complete pre-processed WDC dataset?
I need it to reproduce your work.

[HaoAreYuDong]

If you want to use WDC, it's much easier than using spreadsheet data. we download 50.8 million relational English tables directly from http://webdatacommons.org/webtables/2015/EnglishStatistics.html. Since it does not need table detection and hierarchy extraction because all tables are flat, you can process them with trivial efforts.

[HaoAreYuDong]

A sample C# code:

        List<string> jsonList = new List<string>();
        string writeFileName = "";
        DirectoryInfo folder = new DirectoryInfo("./");
        var fileList = folder.GetFiles("*.table");
        foreach (var file in fileList)
        {
            string fileID = file.Name.Replace(".table", "");
            Console.WriteLine($"Processing with directory: {directory}, file:{fileID}");

            var webTable = JsonConvert.DeserializeObject<WDCTable>(File.ReadAllText(file.FullName));

            if (!webTable.hasHeader)
            {
                Console.WriteLine("Null header");
                continue;
            }

            var table = new WebTable();
            table.Title = webTable.title;
            table.Width = webTable.relation.Count();
            table.Height = webTable.relation.First().Count();

            if (table.Width == 0 || table.Height == 0)
            {
                Console.WriteLine("zero width or height");
                continue;
            }

            table.ID = $"{directory}-{fileID}";
            table.MergedRegions = new List<MergedRegion>();
            table.TopHeaderRowsNumber = 1;
            table.LeftHeaderColumnsNumber = 0;

            table.Texts = new List<List<string>>();
            for (int row = 0; row < table.Height; row++)
            {
                var rowList = new List<string>();
                for (int column = 0; column < table.Width; column++)
                {
                    rowList.Add(webTable.relation[column][row]);
                }
                table.Texts.Add(rowList);
            }

            // top tree root
            table.TopTreeRoot = new HierarchyNode(-1, -1);
            for (int i = 0; i < table.Width; i++)
            {
                table.TopTreeRoot.Cd.Add(new HierarchyNode(0, i));
            }
            // left tree root

            jsonList.Add(JsonConvert.SerializeObject(table));
            writeFileName = table.ID;
        }
        File.WriteAllLines($"{writeFileName}.json", jsonList.ToArray());

[lefterisloukas]

Thanks, could you do the same for the WikiTables dataset too?
Here's the link to the issue: #4