Taming configuration files - macro substitution

So What's the problem?

In the last few blog on global state, I proposed the solution to global state was a server.  I still stand by that, and that concept still needs a bit of fleshing out.  In the end the global state will probably be in a stand alone server connected to by clients using udp or tcp.  Before we get there I will define a server connection library that I can use in a number of servers.

The point is this approach will require late binding, configuration files that depend on the global state server will of necessity not be fully defined until runtime, and the current term parser has no concept of macro substitution.

Extension to term definition

The term definition is extended by the inclusion of a macro definition:

{macro, Name, DataDef} 

which is included in the configuration file as required.

To substitute a macro for a value, the result of the substitution must be the same as if the value were used in its stead, therefore our validation rule for macro substitution is:

maybe_validate(DataDef,{macro,_Name,DataDef},State) ->

    {true,State};

This rule is included in term_defs.erl.

Substitution Code

The code for the substitution is also relatively simple:

-module (substitute).

-export([substitute/2,test/0]).

-type proplist()::[{any(),any()}].

-spec substitute(Data,SubstList) -> OutData when

      Data :: any(),

      SubstList :: proplist(),

      OutData :: any().

substitute(X, SL) when is_list(X) ->

    list_sub(X ,[],SL);

substitute({macro,Name,Def},SL) ->

    macro_sub(Name,Def,SL);

substitute(X, SL) when is_tuple(X) ->

    L = tuple_to_list(X),

    NL = list_sub(L, [], SL),

    list_to_tuple(NL);

substitute(X,_) ->

    X.

list_sub([], Acc, _) ->

    lists:reverse(Acc);

list_sub([H|T], Acc, SL) ->

    list_sub(T,[substitute(H,SL)|Acc],SL).

macro_sub(Name,Def,SL) ->

    {_,V}=proplists:lookup(Name,SL),

    true = term_defs:validate(Def,V),

    V.

Taming Configuration Files - Global State

Configuring for Global State

Warning: This blog is unashamedly Erlang centric.  The issues discussed do exist in other languages, but the solutions here are for Erlangers. 

In general global state is a pain in the arse.   There is no real reason why different processes should not use different databases for example.  As such I avoid global state whenever possible.    There are examples where global state is unavoidable.  In such cases there are advantages to a centralised store for such information.   IPv4 addresses assigned to computers is an example of global state, and without a central store, such as a DHCP server, difficulties with the allocation and management of these cause real problems.

The worked example below shows how configuration files with schema files and scripts to check dependencies can be used to write systems that manage global state and check that no rules are broken prior to deployment.  makefiles can be used to automate the application of the necessary tools and deploy the resulting files.

While it is true that this process could have been done by making a globals database, writing some referential integrity rules and processes could connect to that database and retrieve the data they require, this approach has certain advantages.

1. The result is an OTP application without external dependencies
2. Configuration file changes can be tracked with normal text based source control procedures.  This would not be possible with a traditional database solution.
3. The OTP server has low resource requirements

A worked example

In this case we are concerned with a hypothetical example of a bureau service supplying accounting services.  The accounting system is broken up into modules that communicate via UDP.  Each client has their own set of accounting modules, therefore the pair {client,module} maps to a port.  Port sharing is not allowed thus in relational algebra terms, {client,module} and port are both candidate keys.  Every client and module used to define a port must exist in the client and modules sections respectively.   

The configuration file used in this example follows:

%% File globals.config

%% Good configuration file

%% obeys all the rules

%% comment out one of the module or client lines to generate

%% an error in the related tcp_port definition

{module,point_of_sale,pos}.

{module,stock,stock}.

{module,general_ledger,gl}.

{client,amco,amco}.

{client,bamco,bamco}.

{tcp_port,5000,{amco,point_of_sale}}.

{tcp_port,5001,{amco,stock}}.

{tcp_port,5002,{amco,general_ledger}}.

{tcp_port,5100,{bamco,general_ledger}}.

%% uncomment following line to cause a duplicate port number error

%{tcp_port,5000,{bamco,point_of_sale}}.

In my earlier post I discussed describing Erlang terms in a way that allows them to be parsed and checked.  In the course of this work I did find a bug in the choices construction so if you are following the code get the latest from https://github.com/tonywallace64/erlang_config_tamer.git.

The data definition for this configuration file is:

{list,{options,

[{tuple,[{value,module},{builtin,is_atom},term]},

{tuple,[{value,client},{builtin,is_atom},term]},

{tuple,[{value,tcp_port},{builtin,is_integer},

{tuple,[{builtin,is_atom},{builtin,is_atom}]}]}]}}.

This configuration file can be validated against its data definition by running the config_check script as in the earlier post.  If validated the file globals.config.etf is generated.  This configuration file is loaded into a gen_server and made available to other processes in the system.  This is the usual Erlang/OTP pattern for this type of situation.  Using the emacs editor a boilerplate gen_server is available in erlang mode from the menu: Erlang/skeleton/gen_server.  This skeleton is customised by:

1. Loading globals.config.etf as state in init
2. Writing some query routines

This simple gen_server is available from https://github.com/tonywallace64/global_resources/blob/master/src/globals.erl.  Now that the this gen_server is available a second level of checking can be performed.  Using this server a script to check consistency can be written.  Note that using a genserver in this way implies the need for an extra stage of deployment.  config file to config.etf.  Then check for consistency and finally copy (if passed) into production.  This is easily accomplished using Makefile.  

The consistency checking code follows:

-module(checker).

-export ([main/1]).

main(_) ->

    {ok,Pid} = globals:start_link(),

    Tcp_ports = gen_server:call(Pid,{lookup,{tcp_port,'_','_'}}),

    io:format("tcp_port lookup result: ~p~n",[Tcp_ports]),

    check_ports(Tcp_ports,Pid,gb_sets:empty()),

    file:write_file("config_checked_okay", <<>> ).

check_ports([],_,Acc) ->

    Acc;

check_ports([X={tcp_port,Port,{Client,Module}}|T],Svr,Acc) ->

    io:format("checking ~p",[X]),

    NewSet = gb_sets:insert(Port,Acc),

    [{client,Client,_}] = gen_server:call(Svr,{lookup,{client,Client,'_'}}),

    [{module,Module,_}] = gen_server:call(Svr,{lookup,{module,Module,'_'}}),

    io:format("passed~n"),

    check_ports(T,Svr,NewSet).

Note the main/1 entrypoint.  This makes the code escript compatible and hence easily callable from a Makefile.  In the check_ports function calls are made to the gen_server to retrieve data referred to in the tcp_port entries.  Each lookup should return a list of exactly one entry.  If it does not do so a pattern matching exception is generated and the script aborts.  Similarly gb_sets:insert will generate an exception if the same port number is added more than once.  If check_ports returns without generating an exception the file config_checked_okay is touched for use by make.

Finally, throughout this post reference has been made to make.  Here is a the makefile out of the examples directory.  There are other makefiles here such as one to make the checker script and these are available from github.  The point is make makes the magic happen, so that the config file can be changed, and all the checks and deployments applied from there.

# Author Tony Wallace

EFT=deps/erlang_config_tamer

all: config_checked_okay

config_checked_okay: globals.config.etf ebin/globals.beam

escript scripts/checker/ebin/checker.beam

globals.config.etf: globals.config ebin/globals.def.etf

$(EFT)/config_check globals.config ebin/globals.def.etf

ebin/globals.beam: ../src/globals.erl

erlc -o ebin $< 

ebin/globals.def.etf: src/globals.def $(EFT)/datadef.def

$(EFT)/config_check $< $(EFT)/datadef.def

mv src/*.etf ebin

.PHONY: clean

clean:

rm  ebin/*.beam

rm config_checked_okay

Gospel in 12 steps, step 2 God is real

God is real

Came to believe that a Power greater than ourselves could restore us to sanity.

Those in AA often call this higher power God, as in the serenity prayer:

God, grant me the serenity to accept the things I cannot change,

Courage to change the things I can,

And wisdom to know the difference.

By definition this higher power must be greater than the self, because the self has been unable to stop the drinking or whatever other sin or addiction is involved.   The self is too weak and someone, or something greater is required.

Similarly in Hebrews 11:6 we read:

And without faith it is impossible to please God, because anyone who comes to him must believe that he exists and that he rewards those who earnestly seek him.

At this point many Christians say "The don't believe in Jesus!  The bible says there is no other name under heaven by which man may be saved!" Acts 4:12.  I say to such people, did Abraham, or David know the name of Jesus?

These people remind me of the Pharisees in John 7:52 "They replied, “Are you from Galilee, too? Look into it, and you will find that a prophet does not come out of Galilee.”  In other words God must come on their terms.  In my introduction I spoke of actions speaking louder than words.  A dry drunk is a daily miracle.  Finally I would say, can our puny minds comprehend the Almighty?  Of course not!   Every one of us fails to grasp the eternal one, so if God grants them sobriety or overcomes some other addiction let us rejoice with them.  

Everyone's perception of God is in some way deficient, and so we are all in the same boat.  God understands our weakness, our need for a saviour, someone bigger and stronger than us to stand up for us, a cosmic big brother.  God is pleased when we take our emptiness and failure to him and say, I have tried and failed, I can't do this on my own.

Step 1: Powerlessness

The gift of powerlessness

1. We admitted we were powerless over alcohol—that our lives had become unmanageable.

AA first step

The Parable of the Pharisee and the Tax Collector

9 To some who were confident of their own righteousness and looked down on everyone else, Jesus told this parable: 10 “Two men went up to the temple to pray, one a Pharisee and the other a tax collector. 11 The Pharisee stood by himself and prayed: ‘God, I thank you that I am not like other people—robbers, evildoers, adulterers—or even like this tax collector. 12 I fast twice a week and give a tenth of all I get.’

13 “But the tax collector stood at a distance. He would not even look up to heaven, but beat his breast and said, ‘God, have mercy on me, a sinner.’

14 “I tell you that this man, rather than the other, went home justified before God. 

Luke 18:9-14

(C) Freedom's just another word, for (G) nothing left to lose 

(D) Nothing ain't worth nothing but it's (G) free 

(C) Feeling good was easy Lord when (G) Bobby sang the blues 

(D) Feeling good was good enough for me 

Good enough for me and Bobby Mc(G)Gee

Me and Bobby McGee

Lyrics  Kris Kristofferson and Fred Foster

Powerlessness and pain.  A normal human condition that is for some the beginning of the journey of hope which is the gospel, the good news.  Indeed this powerlessness becomes the start of our desperate search for salvation.  

Consider the beatitudes:

Blessed are the poor in spirit, theirs is the kingdom of heaven.

Out of our desperate spiritual poverty, we cry out, and God hears.

The Gospel of the 12 steps

A personal introduction

When I was a teenager, around 16 or 17 years of age, I used to go and visit a neighbour a couple of doors down the street.  To save time I would cut through the next door neighbour, and I would visit Michael.

I imagine at the time he would have been in his mid forties.  We would sit at his kitchen table have a coffee and talk late into the night.  He would talk to me about life, but mostly he would talk about AA, Alcoholics Anonymous.  

A couple of years later, I sat in the lounge of another man and I became a Christian.

These blog posts are intended to bring some synchronism between the two.  There is no doubt that the history of AA is indebted to Christianity, yet at the same time, I contend, much of the church needs to be learning from AA.

Every dry day for every member of AA is a new miracle, a new testimony to deliverance.  This is salvation.  We have proverbs such as "seeing is believing", "actions speak louder than words", "the proof of the pudding is in the eating".  Jesus himself asked those who witnessed his works to believe his words because of his works.  The man born blind certainly regarded Jesus as a prophet because of his deeds.  

I do remember as a young Christian hearing AA being criticised as not acknowledging Jesus.  I will deal with that in a later post, but apart from that there is little that a Christian could object to in the 12 steps.  Indeed these steps should be part of the life of every Christian.

It is my intention to unpack these steps over the next few weeks.  So much of our organized Christianity is in opposition to the gospel.  Remember Jesus teaching of the Pharaisee and the Tax Collector.  Which do we welcome in our churches, the drunks, the druggies, the smelly, or do we all have to appear to have it all together?  Are our churches safe places or places of hypocracy, judgement and moral superiority.  What would Jesus say about our churches today?  Are we really ready to admit our faults and be real?  Is it safe to do so?  How can we truly become the "Body of Christ"?

I truly believe that we can learn from AA the reality of the saving power of Jesus Christ.  AA has codified the gospel into baby steps for us to follow.

The 12 steps

1. We admitted we were powerless over alcohol—that our lives had become unmanageable.
2. Came to believe that a Power greater than ourselves could restore us to sanity.
3. Made a decision to turn our will and our lives over to the care of God as we understood Him.
4. Made a searching and fearless moral inventory of ourselves.
5. Admitted to God, to ourselves, and to another human being the exact nature of our wrongs.
6. Were entirely ready to have God remove all these defects of character.
7. Humbly asked Him to remove our shortcomings.
8. Made a list of all persons we had harmed, and became willing to make amends to them all.
9. Made direct amends to such people wherever possible, except when to do so would injure them or others.
10. Continued to take personal inventory, and when we were wrong, promptly admitted it.
11. Sought through prayer and meditation to improve our conscious contact with God as we understood Him, praying only for knowledge of His will for us and the power to carry that out.
12. Having had a spiritual awakening as the result of these steps, we tried to carry this message to alcoholics, and to practice these principles in all our affairs.

Taming Configuration Files - Example deployment

Overview

Over the past few weeks, this blog has been on building configuration file checking into the application.  The strategy has been to make a tool that checks the configuration file matches a grammar and to build a way to automate this process with make.  The past two posts have been on matching the grammar, and building the script respectively.

This post builds a simple example which shows this system in use.  A helloworld type program which reads name from a configuration file.

The test

The configuration file hello.conf contains the entry:

{name,"Tony"}.

The checker will confirm that this matches the datadef stored in hello_conf.def which contains:

{property_list,[{reqd,name,{list,{builtin,is_integer}}}]}.

Although it is true that the definition is longer than the data it describes, the definition of a configuration file is an application development activity, not an operational matter.  It is conceivable that definitions will be provided in a machine readable (rather than human readable) form.  Note that in Erlang a string is a list of integers.

The program that reads this configuration file, hello.erl is:

-module (hello).

-export([main/1]).

main(_) ->

    Name = get_name(),

    io:format("Hello ~s~n",[Name]).

get_name() ->

    {ok,Bin} = file:read_file("hello.conf.etf"),

    PL = binary_to_term(Bin),

    proplists:get_value(name,PL).

So this program produces the string "Hello Tony". This program reads the compiled configuration file hello.conf.etf which is produced by the config_check program we built last blog post.

Finally the compilation and test is run by make.  Here is the Makefile:

test: hello.conf.etf ebin/hello.beam

escript ebin/hello.beam

hello.conf.etf:hello_conf.def hello.conf

./config_check hello.conf hello_conf.def

ebin/hello.beam: src/hello.erl

erlc -o ebin src/hello.erl

clean:

rm hello.conf.etf ebin/hello.beam

To run the test type make from the terminal prompt.  To run with a different name, simply edit hello.conf and type make again.

Taming configuration files - making an executable

Overview

In my last blog I showed how to test that an Erlang term matched an arbitrary pattern, including testing for an iolist, a recursive structure.  The objective here is to package this code in an executable that can be deployed in our projects.  The output of a successful check will be a new config file with an ".eft" suffix.  This is to allow tools like make to do consistency checks.

To do this we package this code into an escript.  This requires a short driver program that presents the main/1 interface that escript requires.

The driver program

The name of the escript, and the name of the escript's main module must be the same, and this module must export main/1.  Having decided to name this script "config_check" then the main module bears that name. 

The main/1 entry point takes all of the parameters of the script as a list.  Here is the code.

-module (config_check).

-export([main/1,test/0]).

-purpose (

<<"Write etf version of Config file if valid.",

  "Other requirements are: runable as escript">>).

main([ConfigFile,SchemaFile]) ->

    DataDef = get_schema(SchemaFile),

    {ok,Config}  = file:consult(ConfigFile),

    Result       = term_defs:validate(DataDef,Config),

    write_results(Result,ConfigFile,Config).

write_results(true,ConfigFile,Config) ->

    OutputFile = [ConfigFile,".etf"],

    EtfData = term_to_binary(Config,[]),

    ok = file:write_file(OutputFile,EtfData);

write_results(false,_,_) ->

    ok.

get_schema(SchemaFile) ->

    get_schema_by_ext(filename:extension(SchemaFile),SchemaFile).

get_schema_by_ext(".etf",SchemaFile) ->

    {ok,Etf} = file:readfile(SchemaFile),

    binary_to_term(Etf);

get_schema_by_ext(_,SchemaFile) ->

    {ok,[DataDef]} = file:consult(SchemaFile),

    DataDef.

    

test() ->

    DataDef = "{list,{value,valid}}.",

    DefFile = "main_test.config_def",

    ok = file:write_file(DefFile,DataDef),

    ok = file:write_file("main_test_valid","valid."),

    ok = file:write_file("main_test_invalid","invalid."),

    main(["main_test_valid",DefFile]),

    %% correct behaviour of second test is to throw an exception

    %% catch it and clean up

    catch main(["main_test_invalid",DefFile]),

    true = filelib:is_regular("main_test_valid.etf"),

    false = filelib:is_regular("main_test_invalid.etf"),

    file:delete(DefFile),

    file:delete("main_test_valid"),

    file:delete("main_test_invalid"),

    file:delete("main_test_valid.etf"),

    pass.

    

As usual I have placed a test routine at the end of the module.  Note that as yet there is not test for definition files with ".etf" suffix, this is very naughty.  Notice too that the test DataDef is of the form

{list, {valid,value}}, not {valid,value}.  This is because all files read by file:consult are returned as lists.  In general any configuration file will be a list (or a property list which is still a list).

Making the script

Here I use my own customised version of mad available from https://github.com/tonywallace64/mad.  It does need a rebar.config but that can be empty.  To make the release using the mad executeable:

./mad release script config_check

To automate the process I then made the Makefile:

# Author Tony Wallace
all:config_check

config_check:ebin/config_check.beam ebin/term_defs.beam rebar.config
./mad release script config_check

ebin/%.beam: src/%.erl
erlc -o ebin $< 

clean:
rm config_check ebin/*.beam