[erlang-questions] Testing of nifs

[Kostis Sagonas]

On 10/03/11 10:51, Tim Watson wrote:
> On 2 October 2011 22:27, Dmitry Groshev <lambdadmitry@REDACTED
> <mailto:lambdadmitry@REDACTED>> wrote:
>
>     Well, it's about hidden details. Erlang is pretty straightforward
>     language; C is a lot more tricky to make right. So there is a lot of
>     non obvious bugs that I can't foresee when I write unit tests;
>     PropEr's *random* tests are way better in this sense, because you
>     just describe the domain of your task, not particular cases. You can
>     also test your NIF incrementally, writing property after property
>     (or command after command if there is some state involved).
>     Speaking about my experience — I'm using PropEr to test this library
>     https://github.com/band115/ecirca , that implements mutable circular
>     array in Erlang (all calls to this array are constrained to it's
>     creator, so I hope this mutability can't be a problem). There are no
>     docs right now other than code; I will definitely fix it in a week
>     or two when I'll have more time. I've tried to test it with EUnit
>     when it was just plain array; but then I've added different types of
>     array and realized that I should add like 5 times more code to cover
>     all cases, and there was a thought about adding different types of
>     values (int16/32/64), so I realized than EUnit is definitely not the
>     best tool, just because the space of inputs is so multidimensional
>     (and I really wanted to cover as much of it as I can with tests —
>     Erlang's VM is very fragile to NIFs after all). So I started to dig
>     into PropEr's docs, wrote first commands/checks for statem, compiled
>     a debug version of BEAM, made a habit to attach GDB to the process
>     running tests (this way in case of segfault, and I had a lot of
>     them, I can quickly find where I broke the code) — and right now I
>     can be sure to some degree (which is proportional to amount of tests
>     that you've asked PropEr to do) that this NIFs won't explode in my
>     face tomorrow.
>
>
> That makes perfect sense. I've been struggling in my mind with the best
> ways to test linked-in driver code, which is a little different. ...
>
> At the moment I'm relying on traditional unit testing of the C code
> along with static analysis and integration testing using common_test.

IMO, you should really use PropEr instead of the above. This will take 
you a long way into what you want to eventually achieve.

> I guess what I'd really like is something like PULSE (the randomising
> scheduler employed by QuickCheck) to help me identify where my driver
> code might be locking up the emulator or whatever, but for open source
> projects that's not an option. Currently I rely on load tests in a
> mirror of the target environment*s(, but this isn't really good enough
> because if I move from an 8 core to a 16 core machine, problems can
> mysteriously appear that just didn't show up and they're really hard to
> figure out.

There are better techniques than random-based testing for the types of 
bugs you want to eliminate. One such tool is Concuerror (available from 
https://github.com/mariachris/Concuerror). This is currently work in 
progress and we had not had the time to create a proper homepage for it. 
However, you can take a look at its (really really basic at this point 
-- more to come, hopefully soon) documentation and the examples that its 
distribution contains and also get a good idea of the concurrency errors 
Concuerror can detect and how to use the tool by reading the paper:

     Test-Driven Development of Concurrent Programs using Concuerror
     http://people.inf.ethz.ch/chmaria/Pubs/ERLANG-2011.pdf

The paper is very comprehensive and I strongly recommend it to anyone 
who takes elimination of concurrency errors in Erlang seriously.

Kostis

PS. If your application is open-source, I'll gladly help you in using 
Concuerror, but I suggest we take this off this list, if interested.