Overview
This instalment allows the matching of general Erlang terms including recursive structures such as filename and iolist. Several weeks ago I did some work on matching Erlang terms. This unpublished work has been extended.
This extension involved the storing of parser state, and the change of the validation rules from two parameters to three parameters. Code was broken in this process. This state could have been stored in the process dictionary with a lot less rework, and perhaps where time is money, such as in a commercial environment it would have been done that way. In this rework the existing tests became invaluable in ensuring that there was no loss of quality in the process.
Specifications
As per the earlier work, the intention is to check that an Erlang term conforms to a definition which is also an Erlang term. This comparison is performed by term_defs:validate/2, which takes two parameters, the term specification, and the term respectively. If the two do not match an error is thrown, listing both the rule and the data.Syntactic Conventions
{} - denotes a tuple structure as per normal Erlang
Captialised - denotes a syntactic construction. The details of which are covered in this article.
[] - denotes a list as per normal Erlang
Matching Atomic Values
The test for a given value is done by the {value,Value} term where the corresponding term in the expression to be tested must match Value. Where a type test is required, the {builtin,TypeTest} test is used. TypeTest is any unary function exported from the Erlang module that returns boolean.
Matching from a list of possible values
{options,[Item_Spec]}
In a options construction, one of the Item_Spec's in the list must match. For example: {options,[{value,male},{value,female}]} indicates that the valid contents are one of the set male or female.
Matching a list
{list,TypeTest}
All items within the list must pass the type test. For example to test that all the items in the list are integers use {list,{builtin,is_integer}}.
Defining a pattern
{define, Key, Pattern}
This associates a Pattern with a Key. For example {define int {builtin is_integer}} associates the atom int with the check for an integer. This symbolic referencing allows recursive data patterns to be defined. Defining always returns true.
Matching all items in a list of tests
{match_all,[TypeTest]}
Matching a tuple structure
{tuple, [Test]}
The number of tests in the test list must match the number of positions in the tuple, and each test must be passed for the tuple to be valid. For example {tuple,[{value,employee},{list,{builtin,is_integer}]} will match {employee,"Joe Smith"}.
Matching a property list
{property_list, [ KeyValuePairTest ]}
The KeyValuePairTest is a tuple containing an optionality flag which satisfies the rule {options,[{value,opt},{value,reqd}]}, a KeyName (which is an atom), and a data specification. For every key with the reqd keyword, that item must exist in the data, and its associated value must be valid. For every key with the opt keyword, if it exists then its associated value must be valid. For example, a property list must contain gender, and the value must be either male or female.
{property_list [{reqd,gender,{options[{value,male},{value,female}]}]}
Although the property list construct could have been defined from other rules in this specification, it is a commonly used structure and was therefore given its own abstraction.
A complex example
{match_all,[
{define,iolistmember,{options,[{builtin,is_integer},iolist]}},
{define,iolist,{options,[{builtin,is_binary},{list,iolistmember}]}},
iolist]}
This example tests that data is an iolist. A valid iolist might be [<<"This is a valid ">>,"iolist"].
An iolist is either a binary of a list of integers and io lists. This example shows how recursive structures can be defined using a combination of match_all and define. Notice how it is the last item in the match_all that causes the test to fire, and this item references back to the earlier defines.
The code
-module(term_defs).
-export ([validate/2,test/0]).
-author('Tony Wallace').
-purpose( <<"Confirm that an erlang term matches a specification.",
"The atom datadef matches any data definition",
"Type testing is done by matching the erlang term to {builtin,functionanme}",
"functionname is a unary function exported from the erlang module, for example"
"{builtin,is_integer} will check that the matched term is an integer./n",
"Where an erlang term must match a given term the {value,Term} pattern is used.",
"The {value,Term} construction is valuable where there is a list of valid options",
", for example {options,[{value,option1},{value,option2}]}. In this case the",
"term can match either option1 or option 2./n",
"A list is defined by the construction {list,DataDef}, where each item of the",
"list must conform the the specification in DataDef. For example a list of"
"integers is defined as {list,{builtin,is_integer}}.\n",
"Tuples are matched to {tuple,[DataDef]}. Each term contained within the tuple",
"must match its associated DataDef./n",
"Property lists are given special treatment. A property list is defined its contents",
". The specification is:/n ",
" {property_list,[ ",
" {tuple, ",
" {options,{value,opt},{value,reqd}}, ",
" Key,Specification}]} ",
" opt - this key is optional, reqd this key is required",
"Key - is an erlang term, normally an atom\n ",
"Specification a datadef that that key value must satisfy.">>).
validate(Def,Term) ->
{R,_}=validate(Def,Term,dict:new()),
R.
validate(Def,Term,State) ->
%io:format("validate(~p,~p,~p)~n~n",[Def,Term,State]),
case maybe_validate(Def,Term,State) of
{true,NewState} ->
{true,NewState};
{false,_} ->
not_valid(Def,Term)
end.
%maybe_validate([Def],Term,State) ->
% validate(Def,Term,State);
maybe_validate({define,Key,Value},_Term,State) ->
NewState = dict:store(Key,Value,State),
{true,NewState};
maybe_validate(any,_,S) ->
{true,S};
maybe_validate({property_list,KeyDefs},PL,State) ->
Checked = [pl_entry(KeyDef,PL,State) || KeyDef <- KeyDefs],
R=lists:foldl(fun(true,A) -> A;(_,_)->false end, true, Checked),
{R,State};
maybe_validate({tuple,DefList},Term,S)
when is_tuple(Term) andalso (length(DefList) =:= tuple_size(Term)) ->
TermList = tuple_to_list(Term),
tuple_flds(DefList,TermList,S);
maybe_validate({tuple,_},_,S) ->
%% tuple sizes do not match or Term is not a tuple
{false,S};
maybe_validate({list,TermDef},Term,State)
when is_list(Term) ->
Valid=[validate(TermDef,X,State) || X <- Term],
R=lists:foldl(fun({X,_},A) -> A and X end,true,Valid),
{R,State};
maybe_validate(TermDef={builtin,Atom},Term,State) ->
case catch(apply(erlang,Atom,[Term])) of
{'EXIT',_} ->
not_valid(TermDef,'undefined');
true -> {true,State};
false -> {false,State};
X ->
not_valid(TermDef,{'not_boolean',X})
end;
maybe_validate({options,TermDef},Term,State) when is_list(TermDef) ->
choices(TermDef,Term,State);
maybe_validate({match_all,[]},_,State) ->
{true,State};
maybe_validate({match_all,[H|T]},Term,State) ->
{R1,State1} = validate(H,Term,State),
case R1 of
true ->
validate({match_all,T},Term,State1);
false ->
not_valid(H,Term)
end;
maybe_validate({value,X},X,State) ->
{true,State};
maybe_validate(datadef,{value,_},State) ->
{true,State};
maybe_validate(datadef,{builtin,Fname},State)
when is_atom(Fname)->
EE = erlang:module_info(exports),
case proplists:get_value(Fname,EE) of
1 -> {true,State};
_ -> {false,State}
end;
maybe_validate(datadef,{list,ItemDef},State) ->
validate(datadef,ItemDef,State);
maybe_validate(datadef,{property_list,[{Opt,_KeyName,DataDef}|KeyList]},State) ->
validate([{value,opt},{value,reqd}],Opt,State),
{true,NewState} = validate(datadef,DataDef,State),
maybe_validate(property_list,KeyList,NewState);
maybe_validate(datadef,{property_list,[]},S) -> {true,S};
maybe_validate(datadef,{tuple,DefList},State)
when is_list(DefList)->
Validated = [validate(datadef,X,State) || X <- DefList],
R=lists:foldl(fun({true,_},A) -> A;(_,_) -> false end,true,Validated),
{R,State};
maybe_validate(Key,Term,State) when is_atom(Key) ->
case lookup(Key,State) of
undefined -> {false,State};
Pattern -> validate(Pattern,Term,State)
end;
maybe_validate(_,_,State) ->
{false,State}.
lookup(Key,Dict) ->
lookup2(dict:is_key(Key,Dict),Key,Dict).
lookup2(false,_,_) ->
undefined;
lookup2(true,Key,Dict) ->
dict:fetch(Key,Dict).
pl_entry({Opt,Key,Def},PL,State) ->
case proplists:get_value(Key,PL) of
undefined ->
%% it is valid for an optional key to be undefined
(Opt =:= opt);
Data ->
%% if it exists it must be valid
{true,_}=validate(Def,Data,State),
true
end.
tuple_flds([H1|T1],[H2|T2],State) ->
case validate(H1,H2,State) of
{true,NewState} ->
tuple_flds(T1,T2,NewState);
{false,_S} ->
not_valid(H1,H2)
end;
tuple_flds([],[],State) ->
{true,State}.
choices([H|T],Term,State) ->
%io:format("choices ([~p|~p],~p,~p)~n",[H,T,Term,State]),
case maybe_validate(H,Term,State) of
{true,NewState} ->
{true,NewState};
{false,NewState} ->
choices(T,Term,NewState)
end;
choices([],_,State) ->
{false,State}.
not_valid(Def,Term) ->
throw({invalid,Def,Term}).
test() ->
T=test([
{datadef,{builtin,is_integer},true},
{datadef,{value,value},true},
{datadef,{tuple,[{value,employee}]},true},
{datadef,{tuple,employee},invalid},
{datadef,{list,{builtin,is_integer}},true},
{datadef,{property_list,[]},true},
{{builtin,is_integer},5,true},
{{builtin,is_integer},atom,invalid},
{{options,[{value,option1},{value,option2}]},option2,true},
{{options,[{value,option1},{value,option2}]},option3,invalid},
{{tuple,[{value,employee},{builtin,is_list}]},{employee,"Robert"},true},
{{tuple,[{value,employee},{builtin,is_list}]},{emp,"Robert"},{invalid,{value,employee},emp}},
{{tuple,[{value,employee},{builtin,is_list}]},{employee,"Robert",male},invalid},
{{list,{builtin,is_integer}},[3,4,5],true},
{{list,{builtin,is_integer}},[3,a,5],{invalid,{builtin,is_integer},a}},
{{property_list,[]},[],true},
{{property_list,[{opt,gender,{options,[{value,male},{value,female}]}}]},[],true},
{{property_list,[{opt,gender,{options,[{value,male},{value,female}]}}]},[{gender,male}],true},
{{property_list,[{opt,gender,{options,[{value,male},{value,female}]}}]},[{gender,transgender}],
{invalid,{options,[{value,male},{value,female}]},transgender}},
{{property_list,[{reqd,gender,{options,[{value,male},{value,female}]}}]},[],invalid},
{{match_all,[{define,int,{builtin,is_integer}},int]},5,true},
{{match_all,[{define,int,{builtin,is_integer}},int]},'hello',{invalid,{builtin,is_integer},hello}},
{{match_all,[
{define,iolistmember,{options,[{builtin,is_integer},iolist]}},
{define,iolist,{options,[{builtin,is_binary},{list,iolistmember}]}},
iolist]},
<<"This is a valid iolist">>,true},
{{match_all,[
{define,iolistmember,{options,[{builtin,is_integer},iolist]}},
{define,iolist,{options,[{builtin,is_binary},{list,iolistmember}]}},
iolist]},
[<<"This is a valid ">>,"iolist"],true},
{{match_all,[
{define,iolistmember,{options,[{builtin,is_integer},iolist]}},
{define,iolist,{options,[{builtin,is_binary},{list,iolistmember}]}},
iolist]},
'This is not a valid iolist',
{invalid,{options,[{builtin,is_binary},{list,iolistmember}]},'This is not a valid iolist'}},
{{match_all,[
{define,iolistmember,{options,[{builtin,is_integer},iolist]}},
{define,iolist,{options,[{builtin,is_binary},{list,iolistmember}]}},
iolist]},
"This is a valid iolist",true}
]),
Filtered=[{X,Y} || {X,Y} <- T, Y =/= pass],
io:format("~p~n",[Filtered]).
test(L) when is_list(L) ->
[test1(H) || H <- L].
test1(T={Def,Arg,_R}) ->
ExpectedResult = make_result(T),
case catch(validate(Def,Arg)) of
ExpectedResult -> {T,pass};
Failed -> {T,Failed}
end.
make_result({_,_,true}) ->
true;
make_result({Def,Arg,invalid}) ->
{invalid,Def,Arg};
make_result({_Def,_Arg,Term}) ->
Term.
