About

This page contains results from an ETS benchmark showcasing the performance and scalability of a new ETS ordered_set implementation. The new ETS ordered_set implementation is only enabled when the option {write_concurrency, true} is passed to the ets:new function. The write_concurrency option had no effect on the created table when used in combination with ordered_set before the new implementation was added. The following papers describe the data structure that the new ETS ordered_set implementation is based on (the contention adapting search tree):

A Contention Adapting Approach to Concurrent Ordered Sets. Journal of Parallel and Distributed Computing, 2018. Kjell Winblad and Konstantinos Sagonas. (publisher, preprint)

More Scalable Ordered Set for ETS Using Adaptation. In Thirteenth ACM SIGPLAN workshop on Erlang (2014). Kjell Winblad and Konstantinos Sagonas. (publisher, preprint)

Benchmark Description

The benchmark measures how many ETS operations per second X Erlang processes can perform on a single table. Each of the X processes repeatedly selects an operation to do from a given set of operations. The likelihood that a certain operation will be selected is also given to the benchmark. The table that the processes operate on is prefilled with 500K items before each benchmark run starts. The source code for the benchmark is located in the function ets_SUITE:throughput_benchmark/0 (see "$ERL_TOP/lib/stdlib/test/ets_SUITE.erl"). Below is a list with brief descriptions of the operations:

insert = an ets:insert/2 call with a random item within the key range as the second parameter value
remove = an ets:remove/2 call with a random item within the key range as the second parameter value
lookup = an ets:lookup/2 call with a random item within the key range as the second parameter value
lookup = an ets:lookup/2 call with a random item within the key range as the second parameter value
nextseqX = a sequence of X ets:next/2 calls. The first next call is given a random item within the key range as the second parameter value, and the other next calls are given the result of the previous next call as the second parameter value.
selectAll = an ets:select_count call that counts all the items in the table
partial_selectX = an ets:select_count call that counts all the items within a random range of size X

Benchmark Machine and Erlang Parameters

Machine Configuration

Machine:
Microsoft Azure VM instance: Standard D64s v3 (64 vcpus, 256 GB memory):

2 * Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz (16 cores with hyper-threading)
2 NUMA nodes
32 cores
64 hardware threads
256GB ram

Operating System:

Description:	Ubuntu 18.04.2 LTS
Linux version:	4.18.0-1014-azure

Erlang Parameters

erl +sbt tnnps

Benchmark Results

ETS Benchmark Result Viewer

This page generates graphs from data produced by the ETS benchmark which is defined in the function ets_SUITE:throughput_benchmark/0 (see "$ERL_TOP/lib/stdlib/test/ets_SUITE.erl").

Note that one can paste results from several benchmark runs into the field below. Results from the same scenario but from different benchmark runs will be relabeled and ploted in the same graph automatically. This makes comparisons of different ETS versions easy.

Note also that that lines can be hidden by clicking on the corresponding label.

Paste the generated data in the field below and press the Render button:

# Each instance of the benchmark runs for 9.0 seconds:
# The result of a benchmark instance is presented as a number representing
# the number of operations performed per second:

# To plot graphs for the results below:
# 1. Open "$ERL_TOP/lib/stdlib/test/ets_SUITE_data/visualize_throughput.html" in a web browser
# 2. Copy the lines between "#BENCHMARK STARTED$" and "#BENCHMARK ENDED$" below
# 3. Paste the lines copied in step 2 to the text box in the browser window opened in
#    step 1 and press the Render button

#BENCHMARK STARTED$
Scenario:  50.000000% insert,  50.000000% delete | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[set,public,{write_concurrency,true},{read_concurrency,true}] ; 1155735.777778; 2201059.666667; 3651731.000000; 5691178.555556; 6423040.666667; 1310686.333333; 1380807.111111$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ; 772110.222222; 1329925.777778; 2370828.222222; 4613813.888889; 8752139.222222; 14278293.666667; 20245928.000000$
Scenario:  10.000000% insert,  10.000000% delete,  80.000000% lookup | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[set,public,{write_concurrency,true},{read_concurrency,true}] ; 1150138.888889; 2059612.777778; 3808837.333333; 6884231.555556; 12055328.333333; 10320806.777778; 5278569.000000$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ; 760536.777778; 1358975.222222; 2409625.888889; 4459947.111111; 8889290.888889; 14803124.888889; 21267455.000000$
Scenario:  1.000000% insert,  1.000000% delete,  98.000000% lookup | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[set,public,{write_concurrency,true},{read_concurrency,true}] ; 1120546.888889; 2398521.555556; 4310086.777778; 7931431.777778; 15582126.555556; 24346360.111111; 26480442.111111$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ; 811648.333333; 1530402.444444; 2823342.777778; 5032588.222222; 9790614.333333; 18019552.444444; 25123463.111111$
Scenario:  100.000000% lookup | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[set,public,{write_concurrency,true},{read_concurrency,true}] ; 1250376.555556; 2429999.555556; 4491820.222222; 8492151.222222; 17421838.666667; 33253279.888889; 37278641.666667$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ; 847543.222222; 1646063.777778; 3252466.333333; 6094444.555556; 12261507.888889; 23814455.111111; 30254267.444444$
Scenario:  10.000000% insert,  10.000000% delete,  40.000000% lookup,  40.000000% nextseq10 | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[set,public,{write_concurrency,true},{read_concurrency,true}] ; 488235.333333; 905851.666667; 1594653.777778; 2929720.555556; 5059685.222222; 4813146.111111; 3219361.555556$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ; 392641.444444; 678831.888889; 1267143.333333; 2367472.444444; 4518442.666667; 5579105.888889; 6925171.555556$
Scenario:  10.000000% insert,  10.000000% delete,  40.000000% lookup,  40.000000% nextseq1000 | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[set,public,{write_concurrency,true},{read_concurrency,true}] ; 11030.000000; 20985.666667; 41155.111111; 73049.666667; 142283.444444; 271986.333333; 360942.000000$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ; 8861.000000; 16539.333333; 31978.222222; 55945.222222; 115510.555556; 172976.555556; 254853.555556$
Scenario:  10.000000% insert,  10.000000% delete,  79.000000% lookup,  1.000000% selectAll | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[set,public,{write_concurrency,true},{read_concurrency,true}] ; 2579.888889; 5707.666667; 8932.222222; 16637.111111; 35074.000000; 71081.333333; 94483.777778$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ; 2556.222222; 4669.222222; 10878.000000; 18007.111111; 43347.555556; 72329.888889; 111186.888889$
Scenario:  10.000000% insert,  10.000000% delete,  79.000000% lookup,  1.000000% partial_select1000 | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[set,public,{write_concurrency,true},{read_concurrency,true}] ; 2085.111111; 3461.444444; 7363.333333; 13928.000000; 26136.333333; 52257.111111; 68943.444444$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ; 2333.888889; 4648.555556; 8313.888889; 15676.333333; 30589.555556; 61605.777778; 79648.777778$

#BENCHMARK ENDED$

# Each instance of the benchmark runs for 9.0 seconds:
# The result of a benchmark instance is presented as a number representing
# the number of operations performed per second:

#BENCHMARK STARTED$
Scenario:  50.000000% insert,  50.000000% delete | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[set,public,{write_concurrency,true},{read_concurrency,true}] ; 1301737.666667; 2362403.777778; 4270485.777778; 7077750.111111; 8312926.555556; 1309200.888889; 1382106.333333$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ; 932077.333333; 875888.444444; 719754.777778; 78561.555556; 23988.888889; 20060.111111; 25521.333333$
Scenario:  10.000000% insert,  10.000000% delete,  80.000000% lookup | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[set,public,{write_concurrency,true},{read_concurrency,true}] ; 1275073.444444; 2300453.222222; 4347806.444444; 7899772.777778; 14673021.333333; 11827370.777778; 5072807.444444$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ; 949308.000000; 1126110.555556; 1012920.555556; 869891.222222; 193465.000000; 66914.666667; 78965.777778$
Scenario:  1.000000% insert,  1.000000% delete,  98.000000% lookup | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[set,public,{write_concurrency,true},{read_concurrency,true}] ; 1361108.555556; 2632116.333333; 5049816.111111; 9216342.000000; 17930309.000000; 27635952.555556; 27169074.888889$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ; 997206.333333; 1655454.555556; 2372031.555556; 2446268.444444; 1841565.666667; 348158.555556; 219729.000000$
Scenario:  100.000000% lookup | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[set,public,{write_concurrency,true},{read_concurrency,true}] ; 1418991.888889; 2768023.333333; 5414378.222222; 10028031.444444; 20031773.000000; 38684287.777778; 43822367.111111$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ; 1002037.444444; 1807664.000000; 3296106.111111; 5319455.666667; 6572101.222222; 3939501.111111; 5122319.111111$
Scenario:  10.000000% insert,  10.000000% delete,  40.000000% lookup,  40.000000% nextseq10 | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[set,public,{write_concurrency,true},{read_concurrency,true}] ; 547670.444444; 980981.777778; 1861693.000000; 3318852.555556; 5937286.666667; 6283035.777778; 2681831.444444$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ; 589711.444444; 621680.000000; 494697.555556; 396650.888889; 255573.222222; 47929.666667; 37678.000000$
Scenario:  10.000000% insert,  10.000000% delete,  40.000000% lookup,  40.000000% nextseq1000 | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[set,public,{write_concurrency,true},{read_concurrency,true}] ; 11982.555556; 23696.444444; 46410.555556; 85396.444444; 167251.111111; 313776.777778; 422589.888889$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ; 17213.333333; 12717.444444; 11160.666667; 13139.444444; 13901.888889; 5861.666667; 7607.000000$
Scenario:  10.000000% insert,  10.000000% delete,  79.000000% lookup,  1.000000% selectAll | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[set,public,{write_concurrency,true},{read_concurrency,true}] ; 2648.888889; 5115.777778; 10893.111111; 19332.888889; 41556.777778; 76659.222222; 99505.111111$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ; 2690.444444; 4327.333333; 6163.777778; 8994.777778; 14488.444444; 23740.000000; 15899.222222$
Scenario:  10.000000% insert,  10.000000% delete,  79.000000% lookup,  1.000000% partial_select1000 | Key Range Size: 1000000$
; 1; 2; 4; 8; 16; 32; 64$
[set,public,{write_concurrency,true},{read_concurrency,true}] ; 1996.333333; 4394.666667; 7922.444444; 16244.666667; 31813.888889; 62277.000000; 79323.777778$
[ordered_set,public,{write_concurrency,true},{read_concurrency,true}] ; 2373.777778; 3546.666667; 5710.444444; 9098.555556; 21001.000000; 14214.222222; 16295.444444$

#BENCHMARK ENDED$

Include Throughput Plot
Include % More Throughput Than Worst Plot
Include % Less Throughput Than Best Plot
Bar Plot
Same X Spacing Between Points
Show [ordered_set,public,{write_concurrency,true},{read_concurrency,true}]
Show [set,public,{write_concurrency,true},{read_concurrency,true}]