Author: Björn Gustavsson Status: Final/R15B Implemented in OTP release R15B Type: Standards Track Created: 01-Mar-2011 Erlang-Version: OTP_R15B Post-History: **** EEP 36: Line numbers in exceptions ---- Abstract ======== Extend each entry in the call stack backtrace (hereafter called **stacktrace**) returned from the `erlang:get_stacktrace/0` BIF and from the `catch` operator with filenames and line number information. Specification ============= Currently a stack trace returned from `erlang:get_stacktrace/0` (and the `catch` operator) is a list of three-tuples, where each tuple looks like: {Module,Function,Arity} (In some cases, the third element may be a list of arguments instead of the function arity.) We propose to change each tuple to: {Module,Function,Arity,LocationInfo} `LocationInfo` is a property list (a list of two-tuples) that contains filename and line number information. If there is line number information available, the list will look like: [{file,FilenameString},{line,LineNumber}] The list should be accessed using `proplists:get_value/3` or `lists:keyfind/3`, not by direct matching, since a future release may add more items to the list or change the order. The filename is usually the same as the module with the extension ".erl" added, but if function definitions have been placed in a header file, the filename will be the name of the header file. The filename will also be different if the Erlang source file has been generated by a code generator such as yecc. The line number will never be zero; instead `LocationInfo` will be set to an empty list. The list will be empty if there is no location information available. Here are some reasons that location information may be missing: * The module has been compiled with an older BEAM compiler that does not support generation of line number information. * The module was created by calling `compile:forms/1,2` with forms that did not contain non-zero line numbers and/or filenames. * A parse transform created abstract forms having the line number zero. * The module was created using an alternate compiler that did not provide filenames and/or (non-zero) line numbers. * The line number information may have been stripped from the BEAM file. * The exception occurs in a BIF (implemented in C in the run-time system). Implementation requirements --------------------------- This EEP does not specify exactly how line number information should be implemented, but it does impose some requirements on the implementation: * The presence of line number information should have (virtually) no impact on the execution time for a program if no exceptions occur. In practice, that means that an implementation is *not* allowed to add extra instructions or BIF calls that will be executed when no exception occurs. * Line number information should not be dependent on debug information being present in the BEAM file. * Line number information should be included by default in BEAM files. (There could be options to turn off the inclusion of line number information.) * Loading line number information should be the default. There may be an option to turn off loading of line number information in order to save memory. Example ------- In the examples, we will use the following module: -module(example). -export([m/1]). -include("header.hrl"). m(L) -> {ok,lists:map(fun f/1, L)}. %Line 6 and the header file header.hrl: f(X) -> abs(X) + 1. %Line 2 Using R14B01 to call our example module, we get the following result: 1> example:m([-1,0,1,2]). {ok,[2,1,2,3]} 2> example:m([-1,0,1,2,not_a_number]). ** exception error: bad argument in function abs/1 called as abs(not_a_number) in call from example:f/1 in call from lists:map/2 in call from lists:map/2 in call from example:m/1 3> catch example:m([-1,0,1,2,not_a_number]). {'EXIT',{badarg,[{erlang,abs,[not_a_number]}, {example,f,1}, {lists,map,2}, {lists,map,2}, {example,m,1}, {erl_eval,do_apply,5}, {erl_eval,expr,5}, {shell,exprs,7}]}} In a system with line number information enabled, we get: 1> example:m([-1,0,1,2]). {ok,[2,1,2,3]} 2> example:m([-1,0,1,2,not_a_number]). ** exception error: bad argument in function abs/1 called as abs(not_a_number) in call from example:f/1 (header.hrl, line 2) in call from lists:map/2 (lists.erl, line 948) in call from lists:map/2 (lists.erl, line 948) in call from example:m/1 (example.erl, line 6) 3> catch example:m([-1,0,1,2,not_a_number]). {'EXIT',{badarg,[{erlang,abs,[not_a_number],[]}, {example,f,1,[{file,"header.hrl"},{line,2}]}, {lists,map,2,[{file,"lists.erl"},{line,948}]}, {lists,map,2,[{file,"lists.erl"},{line,948}]}, {example,m,1,[{file,"example.erl"},{line,6}]}, {erl_eval,do_apply,5,[{file,"erl_eval.erl"},{line,482}]}, {erl_eval,expr,5,[{file,"erl_eval.erl"},{line,276}]}, {shell,exprs,7,[{file,"shell.erl"},{line,666}]}]}} If we compile the `example` module using the BEAM compiler in R14B01, there will not be any line number information for that module: 1> example:m([-1,0,1,2,not_a_number]). ** exception error: bad argument in function abs/1 called as abs(not_a_number) in call from example:f/1 in call from lists:map/2 (lists.erl, line 948) in call from lists:map/2 (lists.erl, line 948) in call from example:m/1 2> catch example:m([-1,0,1,2,not_a_number]). {'EXIT',{badarg,[{erlang,abs,[not_a_number],[]}, {example,f,1,[]}, {lists,map,2,[{file,"lists.erl"},{line,948}]}, {lists,map,2,[{file,"lists.erl"},{line,948}]}, {example,m,1,[]}, {erl_eval,do_apply,5,[{file,"erl_eval.erl"},{line,482}]}, {erl_eval,expr,5,[{file,"erl_eval.erl"},{line,276}]}, {shell,exprs,7,[{file,"shell.erl"},{line,666}]}]}} Motivation ========== The lack of line number information in exceptions is a major stumbling block for many beginners, and is a time waster for experienced Erlang programmers. An often repeated piece of advice to mitigate the lack of line number information is to write smaller functions. To some extent, that is good advice, but some functions are most naturally written as a single function with many clauses. One example is the `handle_call/3` callback for a `gen_server` process. Another example is test suites. In a typical test suite, every line tests a condition and can potentially fail. It is not practical to put every line that may fail in a separate function. Test suites based on `common_test` are automatically run through a parse transform that provides line number information when an exception occurs. The parse transform inserts before every line code that saves the current function name and line number in the process dictionary. When an exception occurs, the line number can be retrieved and presented. One problem with this approach is that the test suite will run slower, which can cause test cases to fail if timeouts expire in the system being tested. Another problem is that by default the parse transform is only run on the test modules themselves, and therefore exceptions that occur in other parts of the code (support libraries for testing or the product itself) does not have any line information. Rationale ========= We have chosen to let `erlang:get_stacktrace/0` and the `catch` operator return stacktraces with filename and line number information (instead of introducing a new function called, for example, `erlang:get_full_stacktrace/3`). That means that code that simply passes on the stacktrace (to `erlang:raise/3`) does not need to be updated. For example, the following code that catches an exception, logs it, and pass it on does not need to be updated: try some_call_that_may_fail() catch Class:Reason -> Stk = erlang:get_stacktrace(), log(Class, Reason, Stk), erlang:raise(Class, Reason, Stk) end One the other hand, that means that code that assumes that the stacktrace only may contain three-tuples will no longer work and needs to be updated. There are several reasons for the requirement that the line number information should be loaded by default (rather than ordered by giving an option). * In real systems, code size is usually not an issue since it is overshadowed by the memory used for process heaps, off-heap binaries, and ETS table. Therefore, the 10 percent increase of the code size (as measured in the reference implementation) is not an issue for most users, but the benefit of having line number information is potentially huge. * Newcomers to Erlang have the most need for line number information and they should get it without giving any special option. If an option is needed, questions to the mailing lists about how to find from which source line an exception was caused will continue to waste time. * If an option must be given, even developers that know about it may forget to give it and might therefore end up having to investigate an exception without line number information. (Which may waste a lot of time if the problem is not easily reproduce-able.) Therefore it is better that the developers that cannot afford any increase in the size of the loaded code are the ones that must give an option to turn off loading of line number information. Backwards Compatibility ======================= Applications that examine the stacktrace and assume that it contains three-tuples must be updated. The `erlang:raise/3` BIF still accepts three-tuples (it will translate those to four tuples with an empty list in the fourth element); thus it is not mandatory to update calls to `erlang:raise/3`. Implementation ============== The reference implementation can be fetched from Github like this: git fetch git://github.com/bjorng/otp.git bjorn/line-numbers-in-exceptions Here is an overview of the implementation: The BEAM compiler inserts a `line` instruction before every construct that may generate an exception and before every call that will be included in the stacktrace. (Local tail-recursive calls need no `line` instruction, but external tail-recursive calls need a `line` instruction because they may be calls to BIFs.) The `line` instruction has a single operand, an index into a line number table. The line number table is stored in the "Line" chunk in the BEAM file. The "Line" chunk and line instructions increase the file size of BEAM files by about five percent. The loader will remove the `line` instructions from the code that will be executed, but will remember their location and create a table sorted in address order mapping from program counter to line number information. When a stacktrace needs to be built, the run-time system will do a binary search for the program counter of exception-causing instruction and each continuation pointer. For the benefit of embedded system that run in a very constrained memory space, the run-time system can be started with the '+L' option to disable loading of the line number information. The code will still be about one percent larger than code compiled without line number information, because the compiler was unable to do code sharing optimizations on instructions that cause exceptions (such as the `badmatch` instruction). In the current implementation, the line number information increases the size of the loaded code by roughly ten percent. Copyright ========= This document has been placed in the public domain. [EmacsVar]: <> "Local Variables:" [EmacsVar]: <> "mode: indented-text" [EmacsVar]: <> "indent-tabs-mode: nil" [EmacsVar]: <> "sentence-end-double-space: t" [EmacsVar]: <> "fill-column: 70" [EmacsVar]: <> "coding: utf-8" [EmacsVar]: <> "End:"