Implement a new exercise generator

[gvrooyen]

There is a basic exercise generation script (bin/gen-exercise.sh) which does not yet make use of configlet create to correctly populate the new exercise's configuration files, and add blank files for the stub, tests, and example.

A more comprehensive exercise generator would:

1. Run configlet create to create the skeleton structure of the exercise.
2. Add a very basic solution stub, and replicate it as the initial example in the .meta folder.
3. Generate the stubs for the tests from the exercise's canonical-data.json (and possibly a similar track-specific metadata file; see below).

For the last step, each object in the cases array would correspond to a test. property would be the name of the solution's procedure that should be called (converted from camelCase to snake_case), input would specify the function's parameters and their test values, and expected would specify the expected result for the given input arguments.

The value of property and the keys of input could also be used to generate the procedure stubs in the solution stub.

@BNAndras notes that we probably want to be able to define track-specific tests, and define these in the .meta metadata folder as static definitions. Then it could be possible to regenerate the .odin files from scratch, or just add new tests if either the canonical tests or the track-specific definitions are updated:

The main thing to watch out for would be not to clobber the track-specific tests. My spur-of-the-moment though would be perhaps have an additional file in the .meta folder that contains JSON for the extra tests. Then the generator can append the generated test cases from that file to the end of the upstream test cases.

[glennj]

PR #47 updated the generator to use configlet create and to fetch filenames from .meta/config.json.

It does not create proper test cases though. It very crudely extracts input and expected JSON data from the canonical data, and stashes those into the test suite as strings. Much human intervention is then required.

[rmonnet]

I have had trouble with the test generator on the two exercises I created (High Score and Circular Buffer). I spent some time last night looking at other tracks and trying to come up with an approach. I have a proposal which I posted here for now.

I am not sure of the best approach for us and other to discuss:

1. use this Issue but there is no way to post long descriptions (except may be using github gists)
2. move the discussion to the forum (exercism-support or may be buildind-exercism) to get a wider audience (still no way to post lone descriptions)
3. create a PR with the proposal document I posted to the gist above and use the comment system to go back and forth on the approach. When we agree we can either discard the document (reject the PR) or use it as design doc for the upcoming Odin Test Generator.

@glennj, let me know which approach you want to go with. If 3. I can post the PR and we can take the discussion threre. Also should we invite others to the discussion?

[glennj]

That is a well-written and persuasive document. Another data point: Perl5 is another track that takes the template-per-exercise approach.

I think the forum is probably the best place to discuss it: it's public, and we'll likely get some thoughtful input from other maintainers. I find lengthy reviews tend to get pretty messy unless the PR creator is ruthless about resolving comments. The building category would be best unless we want to get a programming:odin category created for us.

Personally, I don't really feel the need for a generator. I can enumerate some reasons, but I don't want to throw any water on the fire this early. I am happy to support the creation, and will use it if it exists.

[rmonnet]

As I kept looking at the design of an Odin test generator, I had been pondering if it was worthwhile spending a lot of time upfront on building a new test generator. It will be a medium effort mostly because, after doing more research, there is no good templating engine for Odin and one will have to be implemented (or ported from a different language). Also, I have mix feeling about asking the exercise implementer to learn a template language and spend time creating and debugging the template. In my experience, when the current test generator fails, it may be less time fixing the odin test file and exercise stub by hand (using modern IDE and Odin Language Server for support) than learning a template language, building and debugging the template.

I propose to put this issue on the back burner and focus on just building up our portfolio of exercises so we can launch the track. That seems like a better use of our time. We should probably keep the issue open since the current test generator is not doing a good job anyway and may need to be removed completely or just produce empty stubs for the otin test and exercise stub file.

[rmonnet]

Another failure case: When running the current exercise generator on atbash-cipher, it returns empty files for the solution stub and the test file.

[glennj]

Yeah, this is one with nested cases: objects in the canonical json.

[glennj]

In the jq track, I figured this out. I'll be adding a fix in the rational-numbers PR

[rmonnet]

It looks like I need to learn jq :-) I am impressed with what you can do with just bash and jq.