This EEP proposes a new native datatype similar to records.
Native records are tied to a specific module and use the same syntax as the current (tuple-based) records except for declaration.
Native records can never be fully compatible with all uses of the old records. Therefore, to make native records compelling, they should be implemented as efficiently as possible. We believe that a good implementation would make them faster than maps.
The old records (based on tuples) must always remain for backwards compatibility, and for some usages they could still be the best choice.
There is a new reflection API so that the shell and tools can look into any native record.
Example:
-module(average).
-export([start/0]).
%% New record declaration syntax.
-record #state{ values = [] :: list(number()), avg = 0.0 :: float() }.
start() ->
spawn(fun() -> loop(#state{}) end.
loop(State) ->
receive
{get_avg, From} ->
From ! State#state.avg, loop();
{get_values, From} ->
From ! State#state.values, loop(State#state{values = [], avg = 0.0});
{put_value, Value} ->
Values = [Value | State#state.values],
loop(State#state{values = Values, avg = lists:sum(Values) / length(Values)})
end.
Eliminates the need for header files.
The field names are available at runtime (as opposed to tuple records).
When printing native records, fields are printed in definition order (as opposed to the elements of maps, which don’t have any specified order).
More fields can be added to the definition without having to recompile all code that uses the record.
Native records is a real data type, not merely syntatic sugar for tuples.
Access to the elements of a native records can be restricted to the defining module by not exporting the native record. (That applies to access using the match syntax, not to the reflection API.)
We expect that a good implementation will make native records faster than maps.
Replace most tuple-record usages without having to update anything but the declaration. This includes the major parts of the classic record syntax (creating, reading, updating, and matching).
Create something that is useful in building both APIs and keeping internal state (efficient).
Allows IDEs + static analysis tools to easily infer information about the field names.
The following items will never be considered as goals.
element/2 for native records.undefined as default when neither the record definition nor
the record creation provides a value for the field._ wildcard default for assigning a value to any field not explictly
initialized.Here are some features we might consider implementing in a future release.
lists:key*.ets functions.record_info).A native record is a data structure for holding a fixed number of elements in named fields. Similar to functions, native records are defined in a module and can be exported or kept private. A native record definition consists of the name of the native record, a list of type parameters, and the field names.
Formally:
-record #Name(TVar1, ..., TVarN) {
Field1 [= Value1] [:: Type1],
...
FieldN [= ValueN] [:: TypeN]
}.
Name and FieldN need to be atoms. Name is allowed to be used without quotes
when it is a keyword or a variable. That is, div and Tillstånd
are allowed as Name without quoting them, while '_' and '42' must be quoted.
By default a native-record definition is visible to the defining module only. It
is visible to other modules if exported via -export_record() directive. The
export_record directive is similar to -export() and export_type() directives.
Examples:
-module(example).
-export_record([user, pair]).
-record #user{
id = -1 :: integer(),
name :: binary(),
city :: binary()
}.
-record #state{
count
}.
-record #empty{}.
The order of the field names as declared is preserved and part of the native record definition. When printed the fields are printed in the order defined.
There is no defined maximum number of fields in a native record. Because of the need to define all fields in a record, it is very hard to accidentally create a record with one million elements.
-import_record() #
As it can be seen from the next sections, working with native records outside of
the defining module needs using fully qualified names of the native records:
#misc:user{}, … This quickly may become too cumbersome and
#verbose. The
-import_record directive works similar to the -import and -import_type
directives.
Imported native records can be used by their short names.
-module(example2).
-import_record(example1, [user, pair]).
An -import_record() directive can be placed anywhere in the module
(but obviously after -module() attribute). That makes it possible to
replace an include of an header file with record definitions with an
-import_record() directive without having to move it up to near the
top.
From now on let’s employ more precise terminology (when needed).
A native-record definition (or record definition for short) is the definition of a record in some module:
-module(example).
-record #person{name, age}.
A native-record value (or record value for short) is the term created by the following kind of code:
make_person(Name, Age) ->
#person{name=Name, age=Age}.
Most native-records operations can be used either with or without a module name, similar to how function calls can be given with or without module names.
When no module name is given, the operation refers to a record defined in the same module as the operation. This is called a local operation.
Local operations can succeed even if the record being operated on is not exported.
A local creation operation will fail to compile if it uses an unknown record name or a field name not present in the record definition. It always succeeds at runtime. The record definition in the same generation of the code will be used.
Other local record operations (update, matching, and field extraction) never check whether the record is exported, but can fail at runtime for other reasons. (Described in detail later.)
When a module name is given (either explicitly in the operation or
implicitly using -import_record()), this is called an external
operation, even if the module name is that of the module executing
the operation.
An external creation of a record will always use the record defined in the current code version of the module (the version loaded last). In order for it to succeed, the record have to be exported.
Other operations operating on a record value will only succeed if the record was exported when the record value was created. That applies even if the module name of the record value is the same as the module executing the operation.
Anonymous operations makes it possible to operate on record values without specify the record name. (It is not possible to create a record using the anonymous syntax.)
It is not possible to specify a module name for an anonymous operation.
Instead, an anonymous operation is considered a local operation if the module name of the code executing the operation is the same as the record name in the value. When the module names are not the same, it is considered an external operation.
The compiler is invoked on a single module at a time, and can therefore only check local record operations for validity. Other tools, such as Dialyzer, can analyze multiple modules at a time and can warn when an external operation will fail at runtime.
The following expression creates a native record value (in
the same module misc where the native record is defined):
#user{name = ~"John", city = ~"Stockholm"}
The next expression is used to create a native record value outside of the defining module:
#other_module:user{name = ~"John", city = ~"Stockholm"}
(For this to work, the user record must be exported from other_module.)
Specifying the module name of the current module is allowed:
#?MODULE:user{name = ~"John", city = ~"Stockholm"}
This means that the definition of the record from the latest generation of this module is used. (That really only makes a difference if the code creating the record belongs to the old generation.) For this call to succeed, the record must be exported.
For example, the record creation in the foo/0 function will fail at
runtime because the #bar{} record is not exported:
-module(foo).
-record #bar{baz = 1}.
-export([foo/0]).
foo() -> #?MODULE:bar{}.
It is not allowed to create a native-record value in a guard. For example, the following will not compile:
-record #empty{}.
g(R) when #empty{} =:= R -> ok.
A native record can be imported via -import_record directive and
then used by its short name.
-module(example).
-import_record(other_module, [user]).
make_user(Name, City) ->
#user{name = Name, city = City}.
Here is the general syntax for native record creation:
#Name{Field1 = Expr1, ..., FieldN = ExprN}
#Module:Name{Field1 = Expr1, ..., FieldN = ExprN}
Module, Field1, .. FieldN must be atoms. Name must be a atom,
but does not need quotation for keywords and variables. Fields can be
in any order.
Compilation fails with an error if not all Field1, …, FieldN
are unique.
If the definition is local (no module name, neither explictly or
through an -import_record()), compilation will fail if not all field names
are present in the definition.
The record definition in a module will only be used when creating a record value. It will not be used when accessing or updating a record value.
When a native-record value is created, it “captures” key information from the current native-record definition, namely:
The fully qualified name of the native-record (module name and native-record name)
Field names
Whether it is exported (through -export_record)
When a record value is updated or its elements are read, the definition captured when the record value was created is used.
Usually, the definition captured in a record value remains in sync with the definition in the module. However, there are some circumstances in which this is not the case. For example:
If the code for the module has been unloaded.
If a version of the module with a different native record definition has been loaded.
If a record value was sent to another node where the code for the module was not loaded, or the record definition is different in the loaded module on that node.
If no value is provided for a field, and there is a default field
value in the native record definition, the default value is used. If
no value is provided for a field and there is no default field value,
a native record creation fails with a
{novalue,{{Module,Name},FieldName}} error.
In OTP 29, the default value can only be an expression that can be evaluated to a constant at compile-time. Essentially, the allowed expressions are guard expressions, with the following additional restrictions:
No variables.
No calls of any kind.
No creation of any kind of records. Note that the default values for tuple records can contain variables; we don’t want that kind of complication here.
Example:
-record #default{
one = 1,
two = 2*20+1
}.
Reasons for not allowing arbitrary expressions:
Default values require a lookup into the defining module, and the default values can in turn be other native records or even function calls. If we put this together with code upgrade you can get some very strange behaviours where a native record with three fields using the same native record could end up with three different versions of that native record.
It would also be possible to get a cycle where the default value for one native record depends on itself.
Making an efficient implementation allowing arbitrary expressions is probably not possible to achieve for OTP 29.
When creating a native record from a NIF, default values being expressions would not work.
Tuple-based records will initialize a field to undefined if neither
the record definition nor the record creation provides a value.
We consider that to be a mis-feature that will delay the detection of
bugs to either runtime or when Dialyzer is run.
An external native record creation (with a module name given) is validated at runtime against the native record definition. Creation fails in the following cases.
If there is no corresponding native-record definition, creation
fails with a {badrecord,{Module,Name}} exception.
If the native-record definition is not visible at the call site (it
is not exported), creation fails with a {badrecord,{Module,Name}}
exception.
If the native record create expression references the field FN that
is not defined (in the native-record definition), creation fails
with a {badfield,{{Module,Name},FN}} exception.
If no value is provided for a field FN and the native-record
definition has no default value for FN, creation fails with
{novalue,{{Module,Name},FN}} exception.
For a local operation (no module name given), compilation will fail if one or more of the following is true:
There is no record defined in the current module with the given name.
If the creation expression references a field that is not present in the definition for the record.
If the creation expression does not provide a value for a field that has no default value in the definition.
The syntax for accessing native-record fields is as follows:
Expr#Name.Field
Expr#Module:Name.Field
These expressions return the value of the specified field of the native-record value.
When a field from a native-record value is accessed, the captured native-record definition is consulted.
An access operation fails with a {badrecord,Expr} error if:
A module name is specified explicitly or through import_record(),
and the record was not exported at the time this value was created.
Expr does not evaluate to a native-record value of the expected
type #Name or #Module:Name (that is, it is either not a native
record at all or it is another native record).
An access operation fails with a {badfield,{{Module,Name},Field}}
error if:
Field is not defined in the native-record value.The expression can be used in guards — a guard would fail if the corresponding expression raises.
The following syntax allows accessing field Field in any record:
Expr#_.Field
This operation is considered a local operation if the module name of the code executing the operation is the same as the record name in the value. When the module names are not the same, it is considered an external operation.
An access operation fails with a
{badrecord,Expr} error if:
This access operation fails with a {badfield,{{Module,Name},Field}}
error if:
Field is not defined in the native-record value.Here is the syntax for updating native-record values:
Expr#Name{Field1=Expr1, ..., FieldN=ExprN}
Expr#Module:Name{Field1=Expr1, ..., FieldN=ExprN}
Field names must be atoms.
When a native-record value is updated, the native-record definition captured when the record value was created is consulted.
An update operation fails with a {badrecord,Expr} error if:
import_record(),
and the record was not exported at the time this value was created.An update operation fails with a {badfield,FN} error if:
Native-record update expressions are not allowed in guards.
The following syntax allows updating any record that has the given fields:
Expr#_{Field1=Expr1, ..., FieldN=ExprN}
Field names must be atoms.
When a native-record value is updated, its captured definition is consulted, not the current definition in the defining module.
This operation is considered a local operation if the module name of the code executing the operation is the same as the record name in the value. When the module names are not the same, it is considered an external operation.
An update operation fails with a {badrecord,Expr} error if:
The operaation is external and the record value is not marked as exported in its captured definition.
Expr does not evaluate to a native-record value of the expected
type #Name or #Module:Name (that is, it is either not a native
record at all or it is another native record).
An anonymous update operation fails with a
{badfield,{{Module,Name},FN}} error if:
A pattern that matches a certain native-record value is created in the same way as a native-record is created.
The syntax:
#Name{Field1 = Expr1, ..., FieldN = ExprN}
#Module:Name{Field1 = Expr1, ..., FieldN = ExprN}
Here, Expr1, .. ExprN are patterns, and Field names must be atoms.
When a native-record value is matched, its captured native-record definition is consulted.
Pattern matching fails if:
A module name is specified explicitly or through import_record(),
and the record was not exported at the time this value was created.
The pattern references a FieldK and the native-record value does not contain this field.
Note, however, that it is possible to match on the name of a
non-exported record. Thus, if the match_name/1 function in the
following example is called with an instance of record r defined in
some_module, it will succeed even if the record is not exported:
-module(example).
-export([match_name/1]).
match_name(#some_module:r{}) ->
ok.
The following syntax allows matching any record having the named fields:
#_{Field1 = Expr1, ..., FieldN = ExprN}
This operation is considered a local operation if the module name of the code executing the operation is the same as the record name in the value. When the module names are not the same, it is considered an external operation.
Anonymous pattern matching fails if:
The operation is external and the record was not exported at the time this value was created.
The pattern references a FieldK and the native-record value does not contain this field.
Fetching the record index using the #name.field syntax is not
supported, because there is no way it can actually be used, since neither
ETS nor element/2 will work with native records.
is_record/3 #
The existing is_record/3 BIF is overloaded to also accept a native record:
-spec is_record(Term :: dynamic(), Module :: module(), Name :: atom()) -> boolean();
(Term :: dynamic(), Name :: atom(), Arity :: non_neg_integer()) -> boolean().
If Module is a module name and Name is an atom, the predicate
returns true if term Term is a native-record value with the
corresponding native-record name.
This function only checks the module and record name in the captured definition of record. It does not check whether the record was or still is exported.
Example:
-module(misc).
is_user(U) -> is_record(U, some_module, user).
is_record/2 #
The existing is_record/2 function is extended to also work on native
records:
is_record(Term :: dynamic(), Name :: atom()) -> boolean().
Name must be the name of one of the following:
-import_record()When is_record/2 is used in a guard, Name must be a literal atom;
otherwise, there will be a compilation error. There will be a
compilation error if Name is neither the name of a local record nor
an imported native record.
If is_record/2 is used in a function body, Name is allowed to be a
variable.
If Name refers to an imported native record, see the description of
is_record/3 for more details.
Examples:
-module(misc).
-record #user {a,b,c}.
is_user(U) when is_record(U, user) ->
true;
is_user(_U) ->
false.
-module(example).
-import_record(misc, [user/0]).
is_user(U) -> is_record(U, user).
is_record/1 #
-spec is_record(Term :: term()) -> boolean().
is_record(Term) returns true if Term is any native record value.
Native records can be documented just as functions/types/callbacks can
be documented. If you export a record it will be visible and you have
to add -doc false. for it to not be shown.
If a spec, type, callback, or native record refers to an undocumented local native record, the compiler will issue a warning.
When attempting to send a native record to an older node (OTP 28 or earlier), the sender should send a message to the logger process and close the connection. Alternatively, the sender could just send it and the receive node will log an error and disconnect.
With addition of native records the runtime values of different types are now ordered as follows:
number()
< atom()
< reference()
< fun()
< port()
< pid()
< tuple()
< record()
< map()
< []
< [_|_]
< bitstring()
Native-record values are ordered by the following properties in this order:
Note that the default values are only used when creating a record values, so they are never present in a record value.
Comparing the number of fields before comparing the field names is similar to how tuples are compared.
Two records are equal if all of these properties are equal.
A new module records provides functionality for basic runtime
reflection. The purpose of this module is to make it possible to
implement tools such as the Debugger. Its use in application code
(especially creating records) is discouraged.
-module(records).
-type create_options() :: #{is_exported := boolean()}.
-spec get_module(record()) -> module().
-spec get_name(record()) -> atom().
-spec is_exported(record()) -> boolean().
-spec get_field_names(record()) -> [atom()].
-spec get(record(), atom()) -> dynamic().
-spec create(Module :: module(), RecordName :: atom(),
Fields :: [{atom(), dynamic()}],
Options :: create_options()) -> record().
-spec update(Src::record(), Module :: module(), RecordName :: atom(),
FieldsMap :: #{atom() => term()}) -> record().
-spec get_definition(Module :: module(),
RecordName :: atom()) ->
{create_options(),
[{FieldName :: atom(), Default :: dynamic()} |
FieldName :: atom()]}.
Here is an example of how to create a record:
1> Fields = [{z, 3}, {x, 1}, {y, 2}].
2> Options = #{is_exported => true},
3> R = records:create(test, rec, Fields, Options).
#test:rec{z = 3,x = 1,y = 2}
Note that record:create/4 does not reference the record definition
in the given module. The example will work even if module test does
not exist or if it has different definition of record rec.
There is currently no way to restrict the use of the records module,
meaning that it could be used to inspect non-exported records or to
create non-exported native records from any module.
The functions in the records module are BIFs.
Attempting to create a local record values that lacks a definition is a compilation error. So is referring to a non-existing field name in an existing record, and also not providing a value for a field with no default value.
Therefore, there will be three compilation errors for the following module:
-module(compilation_errors).
-record #empty{}.
-record #rec{x}.
create1() ->
#empty{whatever=42}.
create2() ->
#rec{}.
create3() ->
#unknown_record{}.
Those creation operations are always compilation errors because they could never succeed at runtime.
Referring to a record without a local definition is also a compilation error for the other operations. All of the following operations results in compilation errors:
-module(more_compilation_errors).
-record #empty{}.
access(R) ->
R#unknown_record.x.
update(R) ->
R#unknown_record{x=0}.
match(#unknown_record{x=X}) ->
X.
These are errors because without a definition we cannot know whether the operation is on a tuple record or a native record. Operating on an unknown tuple record has always been an error.
Referencing unknown fields in defined native records results in compilation warnings. The reason is that they would succeed at runtime when operating on a record from another version of the same module.
For example, the following operations will all result in a warning:
-module(compilation_warnings).
-record #empty{}.
access(R) ->
R#empty.x.
update(R) ->
R#empty{unknown=42}.
match(#empty{x=X}) ->
X.
The compiler checks neither anonymous operations nor operations on external records.
Here is how printed native-records will look like:
%% native-record
#users:user{id = 1,name = "Alice",city = "London"}
Printing of fields follows the field order. Whether a native-record value is exported is not visible through printing.
External term format is extended to support serialization of native-record values. Here is how a record values is encoded:
67 ‘C’ (1 byte)
Number of fields (4 bytes)
0 for local, 1 for exported (1 byte). The other 7 bits in this byte are reserved for possible future extension, therefore decoding should fail if they are not all zeroes.
The module name (atom)
The record name (atom)
Field names in definition order (atoms)
Values for all fields (any type)
Native records are not tuples:
1> -record #vec{x=0, y=0}.
ok
2> V = #vec{}.
#shell_default:vec{x = 0,y = 0}
3> is_tuple(V).
false
4> element(1, V).
** exception error: bad argument
in function element/2
called as element(1,#shell_default:vec{x = 0,y = 0})
*** argument 2: not a tuple
5> element(x, V).
** exception error: bad argument
in function element/2
called as element(x,#shell_default:vec{x = 0,y = 0})
*** argument 1: not an integer
*** argument 2: not a tuple
Nor are they maps:
6> map_get(x, V).
** exception error: bad map: #shell_default:vec{x = 0,y = 0}
in function map_get/2
called as map_get(x,#shell_default:vec{x = 0,y = 0})
*** argument 2: not a map
7> is_map(V).
false
Here is a short summary of the expected performance characteristics of native records.
OTP 29: Expected to be similar to small maps, probably somewhat slower.
OTP 30: Expected to faster than small maps, approaching the speed of tuple records.
The most straightforward and relatively efficient implementation of native records is to model them on small maps.
That is how they are implemented in PR-10617, intended to be included in Erlang/OTP 29. The field names are kept sorted in one array and the values in another array. When matching or updating a native record, the field names to be matched or updated are also sorted (at load time). This means that any number of elements can be matched or updated in a single linear scan of the sorted field names. The original definition order of the field names is kept in a separate array, which only needs to be consulted when printing or enumerating the field names.
The current implementation of native records does not include all the optimizations applied to maps, so it is expected that native records will be somewhat slower than maps.
For Erlang/OTP 30, we plan to implement native records using “thunks”. At load time, for each match operation, specialized native code will be generated that retrieves the values to be matched directly from BEAM registers and compares them directly with the values in the given native record, without any need to compare field names. Update operations will be handled similarly.
Each such thunk will start with a test that verifies that the native record in question has the same definition as in the loaded code for the module (a cheap pointer comparison). If not (for example, if the native record was created from another instance of the code with a different definition), the match or update operation will fall back to the implementation used for small maps.
It is expected that native record will be faster than small maps, approaching the speed of tuple records.
As previously described, when a native-record value is created, its current definition is captured. As currently implemented, the definition is not actually copied. All record values created from the same definition will refer to the original definition in the loaded code.
However, if the code with the definition is unloaded, the definition will be copied into the actual record, increasing the heap size for process holding the record.
When a native record is re-created from the external term format using
binary_to_term/1 or Erlang distribution, the definition will be shared
if a module with the exact same definition is loaded.
We have deleted parts of the EEP for which we didn’t reach a decision in time for OTP 29. The deleted parts can be found in this branch:
https://github.com/bjorng/eep/tree/bjorn/native-records-future
The definition syntax was chosen because it uses similar syntax for defining a native record and for creating a native-record value.
The new syntax for definining a native record is compatible because it would not compile in previous versions.
Any module that uses one of the following attributes requires updating:
-export_record().-import_record().-native_record().(The native_record() attribute is used in the abstract format internally in the
compiler.)
The type record/0 is now defined, which will result in a warning if an application
has a local definition with the same now.
For code that uses native records, parse transforms or any tools using abstract forms would need to be updated.
Here is the initial implementation of native records:
PR-10617: Implement native records
(Now somewhat out of date, because it was based on a previous version of this EEP.)
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.