Taming Erlang Configuration files

Problem Outline

When building systems we normally have several configuration files.  We have these files for a number of reasons.

• As part of the system boot.  The system may need to access various resources such as databases and webservers in order to start.  These resources in turn need data to be supplied to them for them to establish their function.
• To cope with site specific conditions or business rules.
• To set settings that tend to be static and considered unworthy of (or too sensitive for) a user interface.

Configuration files are an essential part of our delivered systems and are here to stay.

There are several aspects of configuration files that tend to be problematic.  

• They are interpreted.  While it is true that the must conform to an overall syntax to be readable by the system, their contents are read and interpreted by the system in an adhoc and as needed basis.  There tends not to be the static analysis that would be performed on compiled code.
• Their effects are dispersed throughout the system making it often difficult to associate the manifestation of the problem with its cause.

What I desire is a static analysis tool, one that says this is an invalid value, or this value is missing, at the time the configuration file is updated, rather than a strange fault of unknown origin occurring some time later.  Certainly there are tool such as XML parsers that can be used for this: putting configuration data in xml and parsing against a schema.  Once validated the data could become available.

Erlang systems tend to be configured with lists of erlang terms loaded in with file:consult.   This is a system understood and familiar to Erlang Programmers.  Besides is it really a good use of time rewriting existing erlang term configurations into XML?  What is needed is a way to assert certain properties of the configuration file.  By adding a definition file to existing Erlang configuration requirements, existing files can be used without modification, and the schema files are transferable between sites and projects.

Requirements

• Everything is Erlang.  This is an Erlang shop.  Language choice is a strategic decision.
• Do static analysis of configuration files.  Justification is the above section.
• Keep it simple.  Rather a limited but useful tool new than a more sophisticated tool that is never delivered or is difficult to use or unreliable.
• Sensible workflow/tool integration.  Static analysis should be part of deployment.  Tools like make should be able to detect out of date files and act accordingly.
• Erlang configuration files tend to be property lists, which sometimes recursively embed property lists.  The system must be able to check that an arbitrary nested key exists.

Design

1. Every configuration file has a schema entry which defines the definition to which the file conforms.  
2. The definition file is a configuration file with a schema entry also.
3. After checking a non human readable form of the configuration file is generated and it is this version that may be used by applications.
4. By checking file timestamps, tools such as make can determine if a configuration file has been changed and needs rechecking.
5. The Erlang module that provides check and compile facilities exports a "main/1" entrypoint so that is easy to integrate into tools like make using escript.
6. Recursive processing of property lists to check key existence.

The master configuration file is as follows:

File conf_def.conf:

{schema,"conf_def.conf.etf"}.

{required_keys,[schema,required_keys]}.

As we can see this configuration file defines itself and validates against a compiled version of itself.  This is a chicken and egg situation resolved by the method conf:compile/1, which allows a compile without validation.  As we can see the value of required_keys is a list of keys, in this case the two keys that are required are schema and required, so this file conforms with its own definition.

A key can also be a list.  For example a key could be [webserver,[listen_port]] which says there should be a key in the configuration file called webserver, which has a property list for a value, and in that property list should be an entry with the key listen_port.

Code

-module (conf).
-export([main/1,compile/1,check/1,consult/1,checkcompile/1,test/0]).

main([]) ->
    ok;
main([H|T]) ->
    checkcompile(H),
    main(T).

checkcompile(F) ->
    check(F),
    compile(F).

consult(ConfFile) ->
    must_read(ConfFile++".etf").

compile(ConfFile) ->
    {ok,SrcData} = file:consult(ConfFile),
    BinData = erlang:term_to_binary(SrcData),
    FileName = ConfFile++'.etf',
    ok=file:write_file(FileName,BinData).

check(ConfFile) ->
    {ok,ConfData} = file:consult(ConfFile),
    ok=check_key(schema,ConfData),
    Schema = proplists:get_value(schema,ConfData),
    B=must_read(Schema),
    Spec = binary_to_term(B),
    Errors =
[
do_it(X,ConfData) || X <- Spec
],
    ErrorResults =
lists:filter(fun (X) -> ok =/= X end,Errors),  
    Status = (ErrorResults =:= []),
    case Status of
true ->
   ok;
false ->
   KL1 = [atom_to_list(X) || X <- ErrorResults],
   KL2 = lists:foldl(fun(X,Acc) -> X ++ ", " ++ Acc end,[],KL1),
   [$,,KL3] = KL2,
   Msg = "Errors for "++ConfFile++KL3,
   throw(Msg)
    end.

must_read(File) ->
    {A,B} = file:read_file(File),
    ok=match(A,ok,fun() -> ["Failed to read file ",File," reason ",atom_to_list(B)] end),
    B.
   
match(A,A,_) ->
    ok;
match(_,_,B) ->
    throw(iolist_to_binary(B())).

do_it({schema,_},_) ->
    %% already checked
    ok;
do_it({required_keys,KeyList},Data) ->
    MissingKeys = [check_key(X,Data)  || X <- KeyList],
    ErrorResults =
lists:filter(fun (X) -> ok =/= X end,MissingKeys),  
    Status = (ErrorResults =:= []),
    case Status of
true ->
   ok;
false ->
   KL1 = [atom_to_list(X) || X <- MissingKeys],
   KL2 = lists:foldl(fun(X,Acc) -> X ++ ", " ++ Acc end,[],KL1),
   [$,,KL3] = KL2,
   io_lib:format("~nMissing keys:~s~n",
[KL3])
    end.

check_key(Key,Data) when is_atom(Key) ->
    case proplists:is_defined(Key,Data) of
true ->
   ok;
false ->
   Key
     end;

check_key([],_Data) ->
    ok;
check_key(_L=[H|T],Data) ->
    %io:format("Checking key ~p in ~p~n",[_L,Data]),
    case check_key(H,Data) of
ok ->
   SubData = proplists:get_value(H,Data),
   check_key(T,SubData);
K -> K
    end.

test() ->
    WsSchemaString=
"{schema,\"conf_def.conf.etf\"}." ++ [$\r] ++
"{required_keys,[schema,webserver,[webserver,port]]}.",
    ok = file:write_file("webserver_schema.conf",WsSchemaString),    
    WsConfig = 
"{schema, \"webserver_schema.conf.etf\"}." ++ [$\r] ++
"{webserver, [{port,9600}]}.",
    ok = file:write_file("webserver.conf",WsConfig),  
    main(["webserver_schema.conf","webserver.conf"]).