[erlang-questions] term_to_binary and large data structures

[Aaron Seigo]

On 2018-06-27 16:39, zxq9@REDACTED wrote:
> On 2018年6月27日水曜日 16時19分14秒 JST Aaron Seigo wrote:
> If you send huge messages you are killing your heartbeats. That doesn't
> play out very well.

Yes, we've noticed ;)

> Instead, consider opening a separate TCP connection between nodes for
> tranfer of large data. That way heartbeats and small inter-node 
> messages
> don't interfere with your huge bulk message transfers.

That would be fine, and quite doable ... but it means everyone having 
similar issues needs to do the same. Why are we using Erlang, again? 
Yes, for ease of use of things like concurrency and distribution. When 
those things break, the reason to use Erlang evaporates. It would be 
better to address the issues (even if it takes a while and some effort 
to do so!) then to work around it.

That said .. the recent work on top of 21 to allow out-of-order sending 
of messages, including during chunked message transmission, to avoid 
head-of-line blocking is really promising and should, once finalized and 
upstreamed, resolve this particular issue.

> (Yes, this is totally the kind of thing that there should be a library
> for, but it isn't something there is a lot of time for unpaid work on.
> That sucks, but there it is.)

IMHO it should be part of the distribution system itself, one way or the 
other, as using a library to work *around* distribution (and which 
itself would need to follow distribution to be transparent) is a little 
daft.

There are enough of us working on paid time these days to make it 
happen, as long as we know what needs doing and what is acceptable. :)

--
Aaron