[erlang-questions] Hipe and Binary - Bitstring

[Oliver Bollmann]

Hi John,

*problem solved!*

The secret is: 
process_flag(min_heap_size,1024*1024*10),process_flag(*min_bin_vheap_size*,1024*1024*10*10),

with this i get *without native* 1,000,000 steps:

#{gc_major_end => 8,gc_major_start => 8,gc_max_heap_size => 0,gc_minor_end => 85,gc_minor_start => 85}

Performance is 100 time faster, the missing factor 4 comes from hipe itself!

Very nice!

Oliver

On 28.03.19 06:50, Oliver Bollmann wrote:
>
> Hi John,
>
> gc tracer(native) *1,000,000* steps:
>
> #{gc_major_end => 36,gc_major_start => 36,gc_max_heap_size => 0,gc_minor_end => 116,gc_minor_start => 116}
>
> gc tracer(*NOT*  native)*1,000*  steps:
> #{gc_major_end => 35,gc_major_start => 35,gc_max_heap_size => 0,gc_minor_end => 1202,gc_minor_start => 1202}
>
> Oliver
>
>
> On 27.03.19 17:22, John Högberg wrote:
>> Nevermind, I'm blind, I just noticed "UseProf" now. I may need more 
>> coffee :o)
>>
>> It's possible that the native code generates less garbage on the 
>> heap, causing fewer GCs, which will be a lot faster if your processes 
>> have a lot of live data as it won't have to copy it over and over. 
>> Try comparing how many garbage collections the process has gone 
>> through with process_info(Pid, garbage_collection), maybe it will 
>> provide some clue.
>>
>> /John
>>
>> On Wed, 2019-03-27 at 16:31 +0100, John Högberg wrote:
>>> Hi Oliver,
>>>
>>> Have you tried comparing performance without eprof?
>>>
>>> eprof uses tracing to figure out which functions take a long time to 
>>> run, which adds considerable overhead to small functions that are 
>>> repeated extremely often. HiPE doesn't support tracing at all, so 
>>> that overhead simply disappears when the module is native-compiled.
>>>
>>> Regards,
>>> John Högberg
>>>
>>> On Wed, 2019-03-27 at 16:18 +0100, Oliver Bollmann wrote:
>>>>
>>>> Hi John,
>>>>
>>>> indeed, on standalone the factor is about 3.7 only :-(
>>>>
>>>> Attached the module i used. The code is part of: 
>>>> https://gitlab.com/Project-FiFo/DalmatinerDB/bitmap
>>>>
>>>> I wonder, where comes the boost?
>>>>
>>>> Facts: OS OSX 10.14.3(64GB)
>>>>           Erlang 20.3.18,
>>>>           the "boost" module use a lot of process directory (about 
>>>> 10GB, almost of this are binaries!)
>>>>
>>>> Any hints?
>>>>
>>>> Oliver
>>>>
>>>> On 27.03.19 13:04, John Högberg wrote:
>>>>> Hi Oliver,
>>>>>
>>>>> I've tried to reproduce this discrepancy on my machine, but I can 
>>>>> only see a modest difference on latest OTP 21 (the results are in 
>>>>> microseconds):
>>>>>
>>>>> Erlang/OTP 21 [erts-10.3.1] [source] [64-bit] [smp:24:24] [ds:24:24:10] [async-threads:1] [hipe]
>>>>> Eshell V10.3.1  (abort with ^G)
>>>>> 1> c(t, []).
>>>>> {ok,t}
>>>>> 2> t:bench(one).
>>>>> 15957304
>>>>> 3> t:bench(union).
>>>>> 559470
>>>>> 4> c(t, [native]).
>>>>> {ok,t}
>>>>> 5> t:bench(one).
>>>>> 3611371
>>>>> 6> t:bench(union).
>>>>> 501871
>>>>>
>>>>> I've attached the source code I used for this test, am I missing 
>>>>> something?
>>>>>
>>>>> Regards,
>>>>> John Högberg
>>>>>
>>>>> On Wed, 2019-03-27 at 10:09 +0100, Oliver Bollmann wrote:
>>>>>>
>>>>>> I use, with binaries like <<1:1000000>>,
>>>>>>
>>>>>> one(F,<<Size:64/unsigned,Bitmap:Size/bitstring,_/bitstring>>) ->
>>>>>>    one(F,Bitmap,0,[]).
>>>>>> one(F, <<0:1,R/bitstring>>,N,Acc) ->
>>>>>>    one(F,R,N +1,Acc);
>>>>>> one(F, <<1:1,R/bitstring>>,N,Acc) ->
>>>>>>    one(F,R,N +1, [F(N) |Acc]);
>>>>>> one(_, <<>>,_,Acc) ->Acc.
>>>>>> union(<<Size:64/unsigned,L:Size/unsigned,P/bitstring>>,
>>>>>>      <<Size:64/unsigned,R:Size/unsigned,_/bitstring>>) ->
>>>>>>    <<Size:64/unsigned, (L bor R):Size/unsigned,P/bitstring>>.
>>>>>>
>>>>>> and call this functions 1,000,000 times, this takes for 1,000 
>>>>>> calls about 20 minutes,
>>>>>>
>>>>>> if i compile with native -compile([native,{hipe, o2}])it takes 3 
>>>>>> seconds for 1,000 calls, so it is about 400x faster !!
>>>>>>
>>>>>> OS: OSX
>>>>>>
>>>>>> What is the secret?
>>>>>>
>>>>>> -- 
>>>>>> Grüße
>>>>>> Oliver Bollmann
>>>>>> _______________________________________________
>>>>>> erlang-questions mailing list
>>>>>> erlang-questions@REDACTED  <mailto:erlang-questions@REDACTED>
>>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>>
>>>>> _______________________________________________
>>>>> erlang-questions mailing list
>>>>> erlang-questions@REDACTED
>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>> -- 
>>>> Grüße
>>>> Oliver Bollmann
> -- 
> Grüße
> Oliver Bollmann
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

-- 
Grüße
Oliver Bollmann

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20190328/959784eb/attachment.htm>