`Mimic.expect/4` silently fails when mocking a function run with `Task.Supervisor.start_child/3`

[pablocostass]

Summary

If you have an error (e.g.: an assert that failed) inside a Mimic.expect/4 call that happens to mock a function being run with Task.Supervisor.start_child/3, Mimic will silently fail and the test will pass, whereas if the same error happens in any other context, the erorr gets properly raised.

# let's say we have this functio
def identity(x) do
  x
end

# and this code snippet is in a test
{:ok, _pid} = Task.Supervisor.start_link(name: MyApp.TaskSupervisor)
x = 1

Mimic.expect(Mod3, :identity, fn received_x ->
  assert received_x == 2
  received_x
end)

Task.Supervisor.start_child(MyApp.TaskSupervisor, fn -> Mod3.identity(x) end)

I suppose this is due to the fact that Task.Supervisor.start_child/2 works in a fire-and-forget kind of way but I just wanted to double check if this is intended behavior, and if anyone has sensible suggestions to catch these silent errors in tests – as a workaround, for now we are sending messages after the assertions and assert_receiving them.

Environment

$ elixir --version
Erlang/OTP 27 [erts-15.2.2] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

Elixir 1.19.1 (compiled with Erlang/OTP 27)

Steps to reproduce

I created a small repo that exemplifies the issue https://github.com/pablocostass/mimic_minimal_examples

Clone it, and run the tests in the second describe block with

$ mix test test/mimic_issues_test.exs:31
Running ExUnit with seed: 764641, max_cases: 24
Excluding tags: [:test]
Including tags: [location: {"test/mimic_issues_test.exs", 31}]


13:23:01.797 [error] Task #PID<0.196.0> started from #PID<0.192.0> terminating
** (ExUnit.AssertionError)

Assertion with == failed
code:  assert received_x == 2
left:  1
right: 2

    (ex_unit 1.19.1) lib/ex_unit/assertions.ex:454: ExUnit.Assertions.assert/2
    test/mimic_issues_test.exs:47: anonymous fn/1 in MimicIssuesTest."test regular process vs. Task vs. Task.Supervisor properly fails with Task.async/1"/1
    (elixir 1.19.1) lib/task/supervised.ex:105: Task.Supervised.invoke_mfa/2
    (elixir 1.19.1) lib/task/supervised.ex:40: Task.Supervised.reply/4
Function: #Function<6.36367260/0 in MimicIssuesTest."test regular process vs. Task vs. Task.Supervisor properly fails with Task.async/1"/1>
    Args: []


  1) test regular process vs. Task vs. Task.Supervisor properly fails with Task.async/1 (MimicIssuesTest)
     test/mimic_issues_test.exs:43
     ** (EXIT from #PID<0.192.0>) an exception was raised:

          Assertion with == failed
          code:  assert received_x == 2
          left:  1
          right: 2
          stacktrace:
            (ex_unit 1.19.1) lib/ex_unit/assertions.ex:454: ExUnit.Assertions.assert/2
            test/mimic_issues_test.exs:47: anonymous fn/1 in MimicIssuesTest."test regular process vs. Task vs. Task.Supervisor properly fails with Task.async/1"/1
            (elixir 1.19.1) lib/task/supervised.ex:105: Task.Supervised.invoke_mfa/2
            (elixir 1.19.1) lib/task/supervised.ex:40: Task.Supervised.reply/4




  2) test regular process vs. Task vs. Task.Supervisor properly fails when not in a Task (MimicIssuesTest)
     test/mimic_issues_test.exs:32
     Assertion with == failed
     code:  assert received_x == 2
     left:  1
     right: 2
     stacktrace:
       test/mimic_issues_test.exs:36: anonymous fn/1 in MimicIssuesTest."test regular process vs. Task vs. Task.Supervisor properly fails when not in a Task"/1
       test/mimic_issues_test.exs:40: (test)

.
Finished in 0.02 seconds (0.00s async, 0.02s sync)
3 tests, 2 failures (2 excluded)

The last test passes, even though it should fail as the others.

[edgurgel]

Yeah we might be able to report that as part of verify_on_exit 🤔

In the meanwhile you can use Mimic.calls to check if calls were made. Might need to wait first for the task to finish and then run Mimic.calls.

[pablocostass]

Yeah we might be able to report that as part of verify_on_exit 🤔

I haven't read much of Mimic's internals yet but I think we could check if any of the :"$ancestors" are a Task.Supervisor^[1] or if the :initial_call matches what a Task.Supervisor.start_child/{3,5} would result on^[2] when are about to call Mimic.Server.do_apply/5 over here

mimic/lib/mimic/server.ex

Lines 119 to 120 in 16bbc27

	if function_exported?(original_module, fn_name, arity) do
	caller_pids = [self() | Process.get(:"$callers", [])]

---

^[1]: this one is less performant / more cumbersome but probably less prone to breaking if Elixir changes how Task.Supervisor.start_child/{3,5} starts the processes. It would entail checking if any of the :"$ancestors" has something similar in their process dictionary

# this is what's in the process dictionary of a `TaskSupervisor`
dictionary: [
    {:"$initial_call", {:supervisor, Task.Supervisor, 1}},
    ...,
    {Task.Supervisor, {:temporary, 5000}}
  ],

^[2]: easier to fetch, though it relies on Process.info/2, and heavily dependent on the use of Task.Supervised.noreply/4 vs Task.Supervised.reply/{3,4}. If Process.info(self(), :initial_call) matches {Task.Supervised, :noreply, 4} we know the mock is running on a process started by Task.Supervisor.start_child/{3,5}

[edgurgel]

Hmm I would not like to have anything specific to Task or Task.Supervisor inside Mimic tbh as this will be too specific IMO. What if people use spawn fn -> do_something() end etc?

I'm not entirely sure this can be detected by Mimic generally.

Maybe the right approach here is to monitor the task to end, and then use Mimic.calls as I suggested? Not sure

[edgurgel]

One way I was thinking was that expect calls could rescue ExUnit.AssertionError and consider this a failure sending a genserver call to the Mimic.Server so that if verify_on_exit has this error recorded? 🤔