Java 2 Ada - Tag generator2023-05-14T19:02:00+00:00Stephane Carrezurn:md5:d12e23c53b2436d6becce3d51ddbdf38AWAReentrant scanner and parser with Aflex and Ayaccurn:md5:549bf23cdcd37b17183eaafe7543f19c2023-05-14T19:02:00+00:002023-05-14T19:02:00+00:00Stephane CarrezAdalexyaccreleasegeneratorbison
<div class="post-text"><h5>[Aflex and Ayacc](Ada/Aflex-Ayacc-code.jpg)</h5><ol><li><ol><li>What's new in Aflex 1.6</li></ol></li></ol><p>- Support the flex options `%option output`, `%option nooutput`, `%option yywrap`, `%option noinput`,</p><pre><code> `%option noyywrap`, `%option unput`, `%option nounput`, `%option bufsize=NNN` to better control the
generated `_IO` package.
</code></pre>- <a href="Aflex">Aflex</a>(https://github.com/Ada-France/aflex) templates provide more control for tuning the code generation and<pre><code> they are embedded with [Advanced Resource Embedder](https://gitlab.com/stcarrez/resource-embedder)
</code></pre>- Support to define Ada code block in the scanner that is inserted in the generated scanner - New option -P to generate a private Ada package for DFA and IO - New directive `%option reentrant` and `%yyvar` to generate a recursive scanner - New directive `%yydecl` to allow passing parameters to `YYLex`<pre><code> or change the default function name
</code></pre><p>Example of `%option` directives to tell <a href="Aflex">Aflex</a>(https://github.com/Ada-France/aflex) to avoid generating several function or procedures and customize the buffer size.</p><p>```<a href="Ada">Ada</a> %option nounput %option noinput %option nooutput %option noyywrap %option bufsize=1024 ```</p><p>The tool supports some code block injection at various places in the generated scanner. The code block has the following syntax where `<block-name>` is the name of the code block:</p><p>```<a href="Ada">Ada</a> %<block-name> {</p><pre><code> -- Put Ada code here
</code></pre><p>} ```</p><p>The `%yytype` code block can contain type declaration, function and procedure declarations. It is injected within the `YYLex` function in the declaration part. The `%yyinit` code block can contain statements that are executed at beginning of the `YYLex` function. The `%yyaction` code block can contain statements that are executed before running any action. The `%yywrap` code block can contain statements which are executed when the end of current file is reached to start parsing a next input.</p><ol><li><ol><li>What's new in Ayacc 1.4</li></ol></li></ol><p>- Support the Bison `%define variable value` option to configure the parser generator - Support the Bison `%code name { ... }` directive to insert code verbatim into the output parser - Recognize some Bison variables `api.pure`, `api.private`, `parse.error`, `parse.stacksize`,</p><pre><code> `parse.name`, `parse.params`, `parse.yyclearin`, `parse.yyerrok`, `parse.error`
</code></pre>- New option `-S skeleton` to allow using an external skeleton file for the parser generator - <a href="Ayacc">Ayacc</a>(https://github.com/Ada-France/ayacc) templates provide more control for tuning the code generation and<pre><code> they are embedded with [Advanced Resource Embedder](https://gitlab.com/stcarrez/resource-embedder)
</code></pre>- New option `-P` to generate a private Ada package for the tokens package - Improvement to allow passing parameters to `YYParse` for the grammar rules - New `%lex` directive to control the call of `YYLex` function - Fix #6: ayacc gets stuck creating an infinitely large file after encountering a comment in an action<p>The generator supports two code block injections, the first one `decl` is injected in the `YYParse` procedure declaration and the `init` is injected as first statements to be executed only once when the procedure is called. The syntax is borrowed from the Bison parser:</p><p>```<a href="Ada">Ada</a> %code decl {</p><pre><code> -- Put Ada declarations
</code></pre><p>} %code init {</p><pre><code> -- Put Ada statements
</code></pre><p>} ```</p><p>Some other Bison like improvements have been introduced to control the generation of the parser code.</p><p>``` %define parse.error true %define parse.stacksize 256 %define parse.yyclearin false %define parse.yyerrok false %define parse.name MyParser ```</p><ol><li><ol><li>How to use</li></ol></li></ol><p>The easiest way to use <a href="Ayacc">Ayacc</a>(https://github.com/Ada-France/ayacc) and <a href="Aflex">Aflex</a>(https://github.com/Ada-France/aflex) is to use <a href="Alire">Alire</a>(https://github.com/alire-project/alire), get the sources, build them and install them. You can do this as follows:</p><p>``` alr get aflex cd aflex_1.6.0_b3c21d99 alr build alr install alr get ayacc cd ayacc_1.4.0_c06f997f alr build alr install ```</p><ul><li>UPDATE*: the `alr install` command is available only with <a href="Alire">Alire</a>(https://github.com/alire-project/alire) 2.0.</li></ul><p>Using these tools is done in two steps:</p><p>1. a first step to call `aflex` or `ayacc` command with the scanner file or grammar file, 2. a second step to call `gnatchop` to split the generated file in separate Ada files</p><p>For example, with a `calc_lex.l` scanner file, you would use:</p><p>``` aflex calc_lex.l gnatchop -w calc_lex.ada ```</p><p>And with a `calc.y` grammar file:</p><p>``` ayacc calc.y gnatchop -w calc.ada ```</p><p>To know more about how to write a scanner file or grammar file, have a look at <a href="Aflex 1.5 and Ayacc 1.3.0">Aflex 1.5 and Ayacc 1.3.0</a>(https://blog.vacs.fr/vacs/blogs/post.html?post=2021/12/18/Aflex-1.5-and-Ayacc-1.3.0) which explains more into details some of these aspects.</p><ol><li><ol><li>Highlight on reentrancy</li></ol></li></ol><p>By default <a href="Aflex">Aflex</a>(https://github.com/Ada-France/aflex) and <a href="Ayacc">Ayacc</a>(https://github.com/Ada-France/ayacc) generate a scanner and a parser which use global variables declared in a generated Ada package. These global variables contain some state about the scanner such as the current file being scanned. The <a href="Ayacc">Ayacc</a>(https://github.com/Ada-France/ayacc) parser generates on its side two global variables `YYLVal` and `YYVal`.</p><p>Using global variables creates some strong restrictions when using the generated scanner and parser: we can scan and parse only one file at a time. It cannot be used in a multi-thread environment unless the scan and parse is protected from concurrent access. We cannot use easily some grammars that need to recurse and parse another file such as an included file.</p><ol><li><ol><li><ol><li>Reentrant scanner</li></ol></li></ol></li></ol><p>The reentrant scanner is created by using the `-R` option or the `%option reentrant` directive. The scanner will then need a specific declaration with a context parameter that will hold the scanner state and variables. The context parameter has its type generated in the `Lexer_IO` package. The `%yydecl` directive in the scanner file must be used to declare the `YYLex` function with its parameters. By default the name of the context variable is `Context` but you can decide to customize and change it to another name by using the `%yyvar` directive.</p><p>``` %option reentrant %yyvar Context %yydecl function YYLex (Context : in out Lexer_IO.Context_Type) return Token ```</p><p>When the `reentrant` option is activated, <a href="Aflex">Aflex</a>(https://github.com/Ada-France/aflex) will generate a first `Context_Type` limited type in the `Lexer_DFA` package and another one in the `Lexer_IO` package. The generator can probably be improved in the future to provide a single package with a single type declaration. The `Lexer_DFA` package contains the internal data structures for the scanner to maintain its state and the `Lexer_IO` package holds the input file as well as the `YYLVal` and `YYVal` values.</p><ol><li><ol><li><ol><li>Reentrant parser</li></ol></li></ol></li></ol><p>On its side, <a href="Ayacc">Ayacc</a>(https://github.com/Ada-France/ayacc) uses the `YYLVal` and `YYVal` variables. By default, it generates them in the `_tokens` package that contains the list of parser symbols. It must not generate them and it must now use the scanner `Context_Type` to hold them as well as the scanner internal state. The setup requires several steps:</p><p>1. The reentrant parser is activated by using the `%define api.pure`</p><pre><code> directive similar to the [bison %define](https://www.gnu.org/software/bison/manual/html_node/_0025define-Summary.html).
</code></pre><p>2. The `%lex` directive must be used to define how the `YYLex` function must be called since it now has some</p><pre><code> context parameter.
</code></pre><p>3. The scanner context variable must be declared somewhere, either as parameter to the `YYParse`</p><pre><code> procedure or as a local variable to `YYParse`. This is done using the new `%code decl` directive
and allows to customize the local declaration part of the `YYParse` generated procedure.
</code></pre><p>4. We must give visibility of the `YYLVal` and `YYVal` values defined in the scanner context variable.</p><pre><code> Again, we can do this within the `%code decl` directive.
</code></pre><p>A simple reentrant parser could be defined by using:</p><p>```<a href="Ada">Ada</a> %define api.pure true %lex YYLex (Scanner) %code decl {</p><pre><code> Scanner : Lexer_IO.Context_Type;
YYLVal : YYSType renames Scanner.YYLVal;
YYVal : YYSType renames Scanner.YYVal;
</code></pre><p>} ```</p><p>However, this simple form is not really useful as you may need to open the file and setup the scanner to read from it. It is probably better to pass the scanner context as parameter to the `YYParse` procedure. For this, we can use the `%define parse.params` directive to control the procedure parameters. The reentrant parser is declared as follows:</p><p>```<a href="Ada">Ada</a> %lex YYLex (Scanner) %define api.pure true %define parse.params "Scanner : in out Lexer_IO.Context_Type" %code decl {</p><pre><code> YYLVal : YYSType renames Scanner.YYLVal;
YYVal : YYSType renames Scanner.YYVal;
</code></pre><p>} ```</p><p>To use the reentrant parser and scanner, we only need to declare the scanner context, open the file by using the `Lexer_IO.Open_Input` procedure and call the `YYParse` procedure as follows:</p><p>```<a href="Ada">Ada</a></p><pre><code> Scanner : Lexer_IO.Context_Type;
...
Lexer_IO.Open_Input (Scanner, "file-to-scan");
YYParse (Scanner);
</code></pre><p>```</p><ol><li><ol><li><ol><li>Grammar examples:</li></ol></li></ol></li></ol><p>To have a more complete example of a reentrant parser, you may have a look at the following files:</p><ul><li><a href="porion-templates-parser-lexer.l">porion-templates-parser-lexer.l</a>(https://gitlab.com/stcarrez/porion/-/blob/master/src/parser/porion-templates-parser-lexer.l)</li><li><a href="porion-templates-parser-parser.y">porion-templates-parser-parser.y</a>(https://gitlab.com/stcarrez/porion/-/blob/master/src/parser/porion-templates-parser-parser.y)</li></ul></div> Advanced Resource Embedder 1.2.0urn:md5:9d168c833ee907e95a358d21ce9436f82022-01-23T09:04:00+00:002022-01-23T09:04:00+00:00Stephane CarrezAdageneratorCGorelease
<div class="post-text"><h5>[Embedding SQL in binary](Ada/resource-embedder-bin.png)</h5><p>The new release contains the following fixes:</p><p>- Fix Ada generator to be able to use other binary content types</p><pre><code> such as `System.Storage_Elements.Storage_Array`
</code></pre>- Fix Ada generator to escape special characters in strings<p>Since the previous version, Fabien Chouteau asked to be able to use an Ada system type to represent a binary content. This is now possible by using the following XML extract for the description to tell the resource embedder how to integrate the file and generate the Ada source code:</p><p>```XML <package></p><pre><code> <resource name='Resources.Help'
format='binary'
type='access constant System.Storage_Elements.Storage_Array'>
<install mode='copy'>
<fileset dir="help">
<include name="**/*.txt"/>
</fileset>
</install>
</resource>
...
</code></pre><p></package> ```</p><p>With the above description, the Ada code generator produces the following package specification:</p><p>```Ada with System.Storage_Elements; package Resources.Help is</p><pre><code> type Content_Access is access constant System.Storage_Elements.Storage_Array;
type Name_Access is access constant String;
type Name_Array is array (Natural range <>) of Name_Access;
Names : constant Name_Array;
function Get_Content (Name : String)
access constant System.Storage_Elements.Storage_Array;
</code></pre><p>private</p><pre><code> ...
</code></pre><p>end Resources.Help; ```</p><p>This example is available in <a href="Embedding help and documentation in Ada (binary)">Embedding help and documentation in Ada (binary)</a>(https://gitlab.com/stcarrez/resource-embedder/tree/master/examples/ada-help-binary). If you prefer to use `String` instead of a `Storage_Array`, have a look at the <a href="Embedding help and documentation in Ada">Embedding help and documentation in Ada</a>(https://gitlab.com/stcarrez/resource-embedder/tree/master/examples/ada-help). Both examples are similar but they are exposing the file using different Ada types.</p><p>To install the tool, follow the instructions given in the initial announcement: <a href="Advanced Resource Embedder for Ada, C and Go">Advanced Resource Embedder for Ada, C and Go</a>(https://blog.vacs.fr/vacs/blogs/post.html?post=2021/06/11/Advanced-Resource-Embedder).</p><p>If you want to know more about the tool, have a look at its documentation:</p><ul><li><a href="Resource Embedder Guide">Resource Embedder Guide</a>(https://resource-embedder.readthedocs.io/en/latest/) <a href="PDF">PDF</a>(https://gitlab.com/stcarrez/resource-embedder/blob/master/docs/are-book.pdf)</li><li>Man page: <a href="are (1)">are (1)</a>(https://gitlab.com/stcarrez/resource-embedder/blob/master/docs/are.md)</li></ul><p>and if you have ideas for improvements, fork the project and submit a pull request!</p></div> Aflex 1.5 and Ayacc 1.3.0urn:md5:45a1bebacf173ea548b9a06d2555561c2021-12-19T13:03:00+00:002021-12-19T13:03:00+00:00Stephane Carrezreleasegeneratorlexyaccbison
<div class="post-text"><ol><li><ol><li>Aflex Version 1.5.2021</li></ol></li></ol><p>The new release provides the following changes:</p><p>- Fix crash when the scanner file uses characters in range 128..255, - Fixed various compilation warnings, - Use `gprbuild` to build and support `alr`, - Reduced number of style compilation warnings in generated code</p><p>To install Aflex, you can use the <a href="Alire">Alire</a>(https://github.com/alire-project/alire) tool by using:</p><p>```<a href="sh">sh</a></p><pre><code> alr get aflex
cd aflex_1.5.2021_33198b8f
alr build
</code></pre><p>```</p><p>You can also build and install it as easily with the following commands:</p><p>```<a href="sh">sh</a></p><pre><code> git clone https://github.com/Ada-France/aflex.git
cd aflex
make
sudo make install
</code></pre><p>```</p><p>You can also install the following Debian package: <a href="aflex_1.5.2021">aflex_1.5.2021</a>(https://blog.vacs.fr/vacs/debian/ubuntu-focal/aflex_1.5.2021_amd64.html).</p><ol><li><ol><li>Ayacc Version 1.3.0</li></ol></li></ol><p>The Ayacc release provides a little bit more improvements:</p><p>- New option `-C` to disable the generation of `yyclearin` procedure, - New option `-E` to disable the generation of `yyerrok` procedure, - New option `-D` to write the generated files to the directory specified, - New option `-k` to keep the case of grammar symbols, - Fixed various compilation warnings, - Generate constants for shift reduce and goto arrays, - Better strong typing in the generated state tables, - Reduced number of style compilation warnings in generated code</p><p>To install Ayacc with <a href="Alire">Alire</a>(https://github.com/alire-project/alire) tool use:</p><p>```<a href="sh">sh</a></p><pre><code> alr get ayacc
cd ayacc_1.3.0_9b8bf854
alr build
</code></pre><p>```</p><p>You can also build and install it as easily with the following commands:</p><p>```<a href="sh">sh</a></p><pre><code> git clone https://github.com/Ada-France/ayacc.git
cd ayacc
make
sudo make install
</code></pre><p>```</p><p>You can also install the following Debian package: <a href="ayacc_1.3.0">ayacc_1.3.0</a>(https://blog.vacs.fr/vacs/debian/ubuntu-focal/ayacc_1.3.0_amd64.html).</p><ol><li><ol><li>Development with Aflex and Ayacc</li></ol></li></ol><p>Aflex is a tool for generating a scanner that recognizes lexical patterns in text. Ayacc uses a BNF-like specification of a grammar to generate an Ada parser for that grammar. These two tools are tightly coupled together. First the scanner's role is to identify tokens that are used by the grammar file. Second, the parser will use the scanner to ask for tokens as it parses the input content.</p><p>The generated parser provides a main procedure `YYParse` which is the entry point to parse an input content. For this, it calls the `YYLex` function provided by the generated scanner.</p><p>The development model is that you write a scanner file that describes the lexical patterns and the grammar file in the BNF-like specification. Aflex reads the scanner file and generates the scanner. Ayacc reads the grammar file and produces the parser. The scanner and parser are composed of several Ada packages and they are used by each other to exchange some types and operations.</p><h5>[Aflex and Ayacc Development Model](Ada/aflex-ayacc.jpg)</h5><p>Ayacc and Aflex need a number of Ada types for their work and the important types are controlled by the Ayacc grammar. They also share some global variables to hold a token that was parsed or describe the current parser state. The list of possible tokens is generated by Ayacc in the `<Parser>_Tokens` package and represented by the `Token` type.</p><p>```<a href="Ada">Ada</a> package <Parser>_Tokens</p><pre><code> type Token is (T_AND, T_OR, T_STRING, ...);
</code></pre><p>end <Parser>_Tokens; ```</p><p>Aflex will generate the `YYLex` function that returns a `Token` type:</p><p>```<a href="Ada">Ada</a></p><pre><code> function YYLex return <Parser>_Tokens.Token;
</code></pre><p>```</p><p>On its side Ayacc generates the `YYParse` procedure in the generated package body. It is up to you and your grammar to decide whether this procedure must be exported as is or encapsulated to another operation.</p><p>```<a href="Ada">Ada</a></p><pre><code> procedure YYParse;
</code></pre><p>```</p><ol><li><ol><li>Writing a scanner with Aflex</li></ol></li></ol><p>The scanner file describes regular expressions and Ada code which are executed when some input text matches the regular expression. The scanner file contains basically three areas separated by a line containing only `%%`. The global layout is:</p><p>```<a href="C">C</a> definitions %% rules %% user code ```</p><p>The scanner extracts below are taken from a CSS parser written in Ada, the complete scanner file is <a href="css-parser-lexer.l">css-parser-lexer.l</a>(https://github.com/stcarrez/ada-css/blob/master/src/parser/css-parser-lexer.l). I highlight here some interesting parts of that lexer.</p><p>The definitions section contains a set of configuration options that tells the Aflex generator how to generate the final Ada scanner. First, the `%unit` directive allows to control the Ada package name used by the final scanner. The scanner package can be a child package which gives a lot of flexibility and control for the program organization.</p><p>```<a href="C">C</a> %unit CSS.Parser.Lexer %option case-insensitive %option yylineno ```</p><p>The `%option case-insensitive` tells Aflex to ignore case when generating the scanner and the `%option yylineno` instructs it to maintain a variable `yylineno` that indicates the current line number. It also maintains the `yylinecol` variable that indicates the column number of the current token. This is useful to keep track of token position in order to report them when an error occurs.</p><p>Because writing regular expressions is difficult, Aflex allows you to create named definitions where you give a name to a regular expression. A regular expression can use another named definition, hence simplifying the creation of complex regular expressions.</p><p>```<a href="C">C</a> h <a href="0-9a-f">0-9a-f</a> nonascii <a href="240-377">240-377</a> unicode \{h}{1,6}(rn|<a href="trnf">trnf</a>)? escape {unicode}|\<a href="^rnf0-9a-f">^rnf0-9a-f</a> string1 "(<a href="^nrf\"">^nrf\"</a>|\{nl}|{escape})*" string2 '(<a href="^nrf\'">^nrf\'</a>|\{nl}|{escape})*' string {string1}|{string2} nl n|rn|r|f ```</p><p>When all named definitions are written, the next section separated by the `%%` marker defines the rules that describe a pattern and an associated action. The pattern is a regular expression and the action corresponds to some Ada code that is executed. Aflex is not aware of the Ada grammar and does not check its validity. It uses the action code as is and write it in the final Ada file. When an action is complex and must be written in several lines, the action block can be enclosed by `{` and `}` (this syntax comes from the C lexer, the two symbols are of course omitted in the final program).</p><p>The extract below shows several pattern and actions:</p><p>```<a href="C">C</a> %% - return '-'; + return '+'; "and" { return T_AND; } "or" { return T_OR; } "not" { return T_NOT; } {string} { Set_String (YYLVal, YYText, yylineno, yylinecol); return T_STRING; } ```</p><p>Here, when the scanner finds the `-` or `+` symbols, it returns the `-` and `+` token. If it finds the word '`and`', it returns the token `T_AND`.</p><p>The last rule of this example will match a CSS string such as `"red"` or `'blue'` and we will get these results when the `YYText` function is called. The action will execute the `Set_String` procedure whose purpose is to populate the `YYLVal` variable (not shown in the example, and for the curious, that procedure is provided by the private part of the parent package `CSS.Parser`, hence it is visible by the generated scanner package `CSS.Parser.Lexer`).</p><p>The last section of the lexer file contains the user Ada code that will be used for the Ada scanner code generation. This section should contain a definition of the scanner package specification with any necessary `with` units. It must also provide the declaration of the `YYLex` function that should return a `Token` type.</p><p>```<a href="Ada">Ada</a> %% with CSS.Parser.Parser_Tokens; package CSS.Parser.Lexer is</p><pre><code> use CSS.Parser.Parser_Tokens;
</code></pre><pre><code> function YYLex return Token;
</code></pre><p>end CSS.Parser.Lexer;</p><p>with Ada.Text_IO; with CSS.Parser.Lexer_dfa; with CSS.Parser.Lexer_io; package body CSS.Parser.Lexer is</p><pre><code> use CSS.Parser.Lexer_dfa;
use CSS.Parser.Lexer_io;
</code></pre><pre><code> pragma Style_Checks (Off);
pragma Warnings (Off);
</code></pre><ol><li><ol><li></li></ol></li></ol><p>end CSS.Parser.Lexer; ```</p><p>The package body must also be written and it should contain a `##` marker which is the place where the Aflex generator will write the scanner code with the `YYLex` function body.</p><p>Because Aflex does not parse the Ada action code in the scanner file, it is not able to indent correctly the final program. The `pragma Style_Checks (Off);` is useful to remove the indentation warnings that the compiler could emit.</p><p>Aflex also generates a `<Lexer>_io` package and a `<Lexer>_dfa` package which provides helper operations for the generated scanner. The `<Lexer>_dfa` package defines two important functions that give access to the current token as a text.</p><p>```<a href="Ada">Ada</a> package CSS.Parser.Lexer_dfa is</p><pre><code> function YYText return String;
function YYLength return Integer;
</code></pre><p>end CSS.Parser.Lexer_dfa; ```</p><p>For example, if the scanner finds the input string `"red"`, the `YYLex` function will return the `T_STRING` value and the `YYText` function will contain the string `"red"`.</p><p>The `<Lexer>_io` package defines the `Open_Input` procedure that opens the file that must be parsed. The package also defines several other procedures and functions but most of them are dedicated to the scanner.</p><p>```<a href="Ada">Ada</a> package CSS.Parser.Lexer_io is</p><pre><code> procedure Open_Input (Fname : in String);
</code></pre><p>end CSS.Parser.Lexer_io; ```</p><ol><li><ol><li><ol><li>Code generation</li></ol></li></ol></li></ol><p>The Ada scanner is generated by running the `aflex` tool. The `-m` option tells the tool to avoid emitting Ada `with` clauses to the `Text_IO` package, the `-s` option disables the default lex rule that emits an echo of unmatched scanner input to the standard output and the `-L` option disables the `#line` directives.</p><p>```<a href="sh">sh</a></p><pre><code> aflex -ms -L css-parser-lexer.l
gnatchop -w css-parser-lexer.ada
</code></pre><p>```</p><p>The tool generates a single Ada file that contains the specification and the body and it must be split by using `gnatchop` to produce the <a href="css-parser-lexer.ads">css-parser-lexer.ads</a>(https://github.com/stcarrez/ada-css/blob/master/src/parser/css-parser-lexer.ads) specification and <a href="css-parser-lexer.adb">css-parser-lexer.adb</a>(https://github.com/stcarrez/ada-css/blob/master/src/parser/css-parser-lexer.adb) package body.</p><ol><li><ol><li>Writing a parser with Ayacc</li></ol></li></ol><p>Ayacc uses the BNF-like grammar definition to generate the parser written in Ada. The grammar file is similar to a grammar file created for <a href="Yacc">Yacc</a>(https://en.wikipedia.org/wiki/Yacc) or <a href="Bison">Bison</a>(https://www.gnu.org/software/bison/) but it contains Ada code for the grammar actions. The grammar file starts with a set of definitions that enumerates the available tokens and declares the main type to describe a parser state.</p><p>Similar to the scanner file, the `%unit` directive allows to control the name of the generated Ada package. I've found very convenient to provide a parent Ada package that defines various types and operations that the generated parser can use. By using the `%unit` and specifying a child package, the generated parser can easily access those operations and types even if they are declared in the private part.</p><p>```<a href="C">C</a> %unit CSS.Parser.Parser %token T_STRING %token T_AND %token T_OR %token T_NOT ```</p><p>The Ayacc generated parser uses a stack to keep track and maintain the state while parsing a content. Each stack entry represents a grammar rule that was recognized and it is possible to also record some specific information. For this, the declaration part must declare the `YYSType` Ada type. That type describes the state and information about a grammar rule. The parser pushes or pops values on the stack according to the grammar rules that are recognized (after a _shift_ or _reduce_ operation). The `YYSType` cannot be a limited type nor can it be abstract because the value must be copied to or from the `YYLVal` global variable.</p><p>In the following definition, the `YYSType` is in fact declared in the private part of the parent package `CSS.Parser` (again, that type is visible to the parser because it is one of its child package).</p><p>```<a href="Ada">Ada</a> {</p><pre><code> subtype YYSType is CSS.Parser.YYstype;
</code></pre><p>} ```</p><p>After the declaration part, the `%%` separator introduces a list of BNF-like rules. Some grammar rules can be associated with some actions which are executed when the grammar rule is recognized. With Ayacc, the action is an Ada code which can contain `$n` constructs which give access to values of the rule elements. In fact, the `$n` values refers to values in the parser stack and the `$$` value is the `YYVal` variable. When an action is executed, this is known as a _reduce_ action and all the rule elements are replaced by the `$$` result.</p><p>```<a href="Ada">Ada</a> %% expr :</p><pre><code> expr operator term
{ CSS.Parser.Set_Expr ($$, $1, $3); }
|
expr term
{ CSS.Parser.Set_Expr ($$, $1, $2); }
|
term
;
</code></pre><p>```</p><p>In this grammar rule extract, the `$$` refers to the current rule value (`YYVal`), the `$1` contains the value of the first element and so on. It is not possible to exceed the number of elements described by the rule. These values are of type `YYSType`. When the rule action is finished, the elements matched by the rule are removed from the stack and the current value `$$` (`YYVal`) is pushed on top of the stack.</p><p>It is possible to write complex action code in the grammar file but this is not very convenient. In many cases, it is easier to provide procedures or functions that the generated parser can use for the action to perform its work.</p><p>After the grammar rules, a last `%%` separator introduces the final Ada code used by the parser generator. That last section must contain the `##` marker which represents the place where Ayacc will emit the body of the parser. The Ada code should also declare the `yyerror` procedure that is called by the generated parser when a syntax error is detected.</p><p>Below is a partial example of such final Ada code:</p><p>```<a href="Ada">Ada</a> %% with CSS.Core.Sheets; package CSS.Parser.Parser is</p><pre><code> Error_Count : Natural := 0;
</code></pre><pre><code> function Parse (Content : in String;
Document : in CSS.Core.Sheets.CSSStylesheet_Access) return Integer;
</code></pre><p>end CSS.Parser.Parser;</p><p>with CSS.Parser.Parser_Goto; with CSS.Parser.Parser_Tokens; with CSS.Parser.Parser_Shift_Reduce; with CSS.Parser.Lexer_io; with CSS.Parser.Lexer; with CSS.Parser.Lexer_dfa; with Ada.Text_IO; package body CSS.Parser.Parser is</p><pre><code> procedure YYParse;
</code></pre><pre><code> procedure yyerror (Message : in String := "syntax error");
</code></pre><pre><code> procedure yyerror (Message : in String := "syntax error") is
begin
Error_Count := Error_Count + 1;
end yyerror;
</code></pre><pre><code> function Parse (Content : in String;
Document : in CSS.Core.Sheets.CSSStylesheet_Access) return Integer is
begin
Error_Count := 0;
CSS.Parser.Lexer_Dfa.yylineno := 1;
CSS.Parser.Lexer_Dfa.yylinecol := 1;
CSS.Parser.Lexer_IO.Open_Input (Content);
CSS.Parser.Parser.Document := Document;
YYParse;
return Error_Count;
end Parse;
</code></pre><ol><li><ol><li></li></ol></li></ol><p>end CSS.Parser.Parser; ```</p><p>To prepare the scanner to read a content, you will have to call the `Open_Input` procedure with the file name as parameter. Then, all you have to do is call the `YYParse` procedure that will call the `YYLex` function and proceed with the scanning and parsing of the input file.</p><ol><li><ol><li><ol><li>Code generation</li></ol></li></ol></li></ol><p>The Ada parser is generated by running the `ayacc` tool. The `-n` option controls the size of the parser stack (the default is 8192). The `-k` option tells the generator to keep the case of tokens as they are written in the grammar file (the default is that it converts the token using mixed case). The `-s` option prints some statistics about the grammar. This is useful to understand and check the shift/reduce and reduce/reduce conflicts that could arise in the grammar. The `-e` option controls the extension of the generated file.</p><p>```<a href="sh">sh</a></p><pre><code> ayacc -n 256 -k -s -e .ada css-parser-parser.y
gnatchop -w css-parser-parser.ada
</code></pre><p>```</p><p>The tool generates a single Ada file that contains the specification and the body and it must be split by using `gnatchop` to produce the <a href="css-parser-parser.ads">css-parser-parser.ads</a>(https://github.com/stcarrez/ada-css/blob/master/src/parser/css-parser-parser.ads) specification and <a href="css-parser-parser.adb">css-parser-parser.adb</a>(https://github.com/stcarrez/ada-css/blob/master/src/parser/css-parser-parser.adb) package body.</p><ol><li><ol><li>Going further</li></ol></li></ol><p>Both tools are very close to the lex/flex and yacc/bison Unix tools which means that most tips and documentation that you will find on Lex and Yacc are common and applicable to Aflex and Ayacc. Writing the grammar file is probably the most difficult task and the <a href="GNU Bison documentation">GNU Bison documentation</a>(https://www.gnu.org/software/bison/manual/html_node/index.html) seriously helps in that task.</p><p>Looking at how scanner and grammar files are written in some other projects can help. Below is a non exhaustive list of scanner and grammar files that can be processed by Aflex and Ayacc:</p><ol><li><ol><li><ol><li>Scanner examples</li></ol></li></ol></li></ol><ul><li>Aflex example <a href="example.l">example.l</a>(https://github.com/Ada-France/aflex/blob/master/doc/example.l)</li><li>Aflex example <a href="test95.l">test95.l</a>(https://github.com/Ada-France/aflex/blob/master/doc/test95.l)</li><li>Aflex example <a href="options.l">options.l</a>(https://github.com/Ada-France/aflex/blob/master/doc/options.l)</li><li>Ayacc calc example <a href="calc_lex.l">calc_lex.l</a>(https://github.com/Ada-France/ayacc/blob/master/examples/calc/calc_lex.l)</li><li>Ayacc Ada83 scanner example <a href="ada_lex.l">ada_lex.l</a>(https://github.com/Ada-France/ayacc/blob/master/examples/ada_parser/ada_lex.l)</li><li>Ada CSS, CSS 3 scanner <a href="css-parser-lexer.l">css-parser-lexer.l</a>(https://github.com/stcarrez/ada-css/blob/master/src/parser/css-parser-lexer.l)</li><li>Ada CSS, CSS 3 definition rule scanner <a href="css-analysis-parser-lexer.l">css-analysis-parser-lexer.l</a>(https://github.com/stcarrez/ada-css/blob/master/tools/parser/css-analysis-parser-lexer.l)</li><li>Memory Analysis Tool, expression scanner <a href="mat-expressions-lexer.l">mat-expressions-lexer.l</a>(https://github.com/stcarrez/mat/blob/master/mat/src/parser/mat-expressions-lexer.l)</li></ul><ol><li><ol><li><ol><li>Grammar examples</li></ol></li></ol></li></ol><ul><li>Ayacc calc example <a href="calc.y">calc.y</a>(https://github.com/Ada-France/ayacc/blob/master/examples/calc/calc.y)</li><li>Ayacc Ada83 grammar example <a href="ada.y">ada.y</a>(https://github.com/Ada-France/ayacc/blob/master/examples/ada_parser/ada.y)</li><li>Ada CSS, CSS 3 parser <a href="css-parser-parser.y">css-parser-parser.y</a>(https://github.com/stcarrez/ada-css/blob/master/src/parser/css-parser-parser.y)</li><li>Ada CSS, CSS 3 definition rule parser <a href="css-analysis-parser-parser.y">css-analysis-parser-parser.y</a>(https://github.com/stcarrez/ada-css/blob/master/tools/parser/css-analysis-parser-parser.y)</li><li>Memory Analysis Tool, expression parser <a href="mat-expressions-parser.y">mat-expressions-parser.y</a>(https://github.com/stcarrez/mat/blob/master/mat/src/parser/mat-expressions-parser.y)</li></ul></div> Advanced Resource Embedder 1.1.0urn:md5:2087a0d7482b1a0f7a7ef5ac368f31792021-07-04T16:52:00+00:002021-07-04T16:52:00+00:00Stephane CarrezAdageneratorCGo
<div class="post-text"><p>The resource embedder allows to embed files in binaries by producing C, Ada or Go source files that contain the original files. The first version of the tool was representing the file content as an array of bytes. In Ada, they are accessed through an `Ada.Streams.Stream_Element_Array` which is not easy to use when you need the content as a string.</p><p>The new release introduces the customization of the format for each resource. The format is controlled through the XML description by the `format` attribute. The following data formats are supported:</p><ul><li>`binary` format provides the file content as a binary data,</li><li>`string` format provides the file content as string,</li><li>`lines` format splits the content in several lines and according to a set of customizable rules.</li></ul><p>With the `string` format, the Ada code generator can generate the following function:</p><p>```</p><pre><code> function Get_Content (Name : in String)
return access constant String;
</code></pre><p>```</p><p>The `lines` format tells the code generator to represent the content as an array of separate lines. For this integration, some control is available to indicate how the content must be split and optionally apply some filter on the input content. These controls are made within the XML description by using the `line-separator` and `line-filter` description: The `line-separator` indicates the characters that represent a line separation. There can be several `line-separator` definition. The `line-filter` defines a regular expression that when matched must be replaced by an empty string or a specified content. The `line-filter` are applied in the order of the XML definition.</p><p>The example below is intended to integrate an SQL scripts with:</p><ul><li>a separate line for each SQL statement,</li><li>remove spurious empty lines and SQL comments.</li></ul><p>The SQL statements are separated by `;` (semi-colon) and the `line-separator` indicates to split lines on that character. By splitting on the `;` we allow to have an SQL statement on multiple lines in the original SQL source file.</p><p>```XML <package></p><pre><code> <resource name='Scripts'
format='lines'
type='access constant String'>
<line-separator>;</line-separator>
</code></pre><pre><code> <!-- Remove new lines -->
<line-filter>[\r\n]</line-filter>
</code></pre><pre><code> <!-- Remove C comments -->
<line-filter>/\*[^/]*\*/</line-filter>
</code></pre><pre><code> <!-- Remove contiguous spaces after C comments removal -->
<line-filter replace=' '>[ \t][ \t]+</line-filter>
</code></pre><pre><code> <install mode='copy' strip-extension='yes'>
<fileset dir="sql">
<include name="**/*.sql"/>
</fileset>
</install>
</resource>
</code></pre><p></package> ```</p><p>Then the first `line-filter` will remove the `r` and `n` characters.</p><p>The regular expression `/\*[./]*\*/` matches a C style comment and remove it.</p><p>The last `line-filter` replaces multiple tabs and spaces by a single occurrence.</p><p>Below is an example of generated code with the above resource description. Each file is accessed through a separate variable whose name is built from the base name of the original file.</p><h5>[Resource Embedder Overview](Ada/are-scripts.png)</h5><p>The command used to generate this code was:</p><p>``` are <del>lang=Ada -o src </del>var-access <del>content-only </del>rule=package.xml . ```</p><p>and the `sql` directory contains only two files: `create-database.sql` and `drop-database.sql`.</p><p>The complete example is available for two languages:</p><ul><li><a href="Embedding SQL scripts in Ada and mapping them in array of String">Embedding SQL scripts in Ada and mapping them in array of String</a>(https://gitlab.com/stcarrez/resource-embedder/tree/master/examples/ada-lines)</li><li><a href="Embedding SQL scripts in C and mapping them in array of String">Embedding SQL scripts in C and mapping them in array of String</a>(https://gitlab.com/stcarrez/resource-embedder/tree/master/examples/c-lines)</li></ul><p>To install the tool, follow the instructions given in the initial announcement: <a href="Advanced Resource Embedder for Ada, C and Go">Advanced Resource Embedder for Ada, C and Go</a>(https://blog.vacs.fr/vacs/blogs/post.html?post=2021/06/11/Advanced-Resource-Embedder).</p><p>If you want to know more about the tool, have a look at its documentation:</p><ul><li><a href="Resource Embedder Guide">Resource Embedder Guide</a>(https://resource-embedder.readthedocs.io/en/latest/) <a href="PDF">PDF</a>(https://gitlab.com/stcarrez/resource-embedder/blob/master/docs/are-book.pdf)</li><li>Man page: <a href="are (1)">are (1)</a>(https://gitlab.com/stcarrez/resource-embedder/blob/master/docs/are.md)</li></ul><p>and if you have ideas for improvements, fork the project and submit a pull request!</p></div> Advanced Resource Embedder for Ada, C and Gourn:md5:4031b79fd70c34caaaf9239de5fa43e12021-06-11T12:43:00+00:002021-06-11T12:43:00+00:00Stephane CarrezAdageneratorCGo
<div class="post-text"><p>The tool is able to apply some transformations on the collected files:</p><ul><li>it can run a Javascript minifier such as `closure`,</li><li>it can compress CSS files by running `yui-compressor`,</li><li>it can compress files by running `gzip` or another compression tool.</li></ul><p>Once these transformations are executed, it invokes a target generator to produce a source file either in C, Ada or Go language. The generated source file can then be used in the final program and taken into account during the compilation process of that program. At the end, the binary will contain the embedded files with their optional transformations.</p><h5>[Resource Embedder Overview](Ada/resource-embedder.png)</h5><p>The process to use ARE is simple:</p><ul><li>You describe the resources that you want to embed.</li><pre><code> The description is either made on command line arguments or by writing an XML file.
The XML description gives more flexibility as it allows to define a transformation rule that
must be executed on the original file before being embedded. This allows to minify a Javascript
or CSS file, compress some files and even encrypt a file before its integration.
</code></pre><li>You run the ARE command with your target language and rule description and you give the tool</li><pre><code> a list of directories that must be scanned to identify the files that must be collected.
The ARE tool scan the directories according to the patterns that you have given either on
the command line or in the XML rule description. After identifying the files, the tool applies
the rules and execute the transformations.
The ARE tool then invokes the target language generator that writes one or several files depending
on the list of resources.
</code></pre><li>Once the files are generated, you use them in your program and add them in your build process</li><pre><code> as they are now part of your sources. After building your program, it now embeds the
resource files that were collected and optionally transformed.
</code></pre></ul><ol><li><ol><li>Defining resources to embed</li></ol></li></ol><p>The first step is to write some `package.xml` file which describes a list of resources with their content. The root XML element is *`package`* and each resource is described by a `resource` XML element. The resource is assigned a name that will be used by the code generator. The C generator will use the name as a prefix for the C function and C type declaration. The Ada and Go generator will use that name for the Ada or Go package.</p><p>The following resource definition declares the `Help` resource. It contains an installation rule that will copy the files under the `help` directory in the resource set. Only files matching the `.txt` pattern will be taken into account.</p><p>```XML <package></p><pre><code> <resource name='Help'>
<install mode='copy'>
<fileset dir="help">
<include name="**/*.txt"/>
</fileset>
</install>
</resource>
...
</code></pre><p></package> ```</p><p>The next resource definition will run an external program to get the content that must be embedded. The `man` directory is scanned and it will execute the command `man #{name}` on each filename found. That directory contains the empty files `ls`, `pwd` and `sh` and this will run and embed the man page for these commands.</p><p>```XML <package></p><pre><code> ...
<resource name='Man'>
<install mode='exec'>
<command output='#{dst}'>man #{name}</command>
<fileset dir="man">
<include name="*"/>
</fileset>
</install>
</resource>
</code></pre><p></package> ```</p><p>You may run other commands such as:</p><p>``` <command>closure <del>charset UTF-8 #{src} </del>js_output_file #{dst}</command> <command>yui-compressor <del>type css </del>charset UTF-8 -o #{dst} #{src}</command> <command output="#{dst}">gzip --no-name -c #{src}</command> ```</p><ol><li><ol><li>Running the Advanced Resource Embedder</li></ol></li></ol><p>The tool has several options that allows you to control the target code generator and tune the generation according to your needs. Assuming that the files to embed are located in the current directory, you would use the following command to generate C source files in the `src` directory:</p><p>``` are <del>lang=c -o src </del>rule=package.xml <del>list-access </del>name-access . ```</p><p>For C, this would generate a `src/man.h`, `src/man.c`, `src/help.h` and `src/help.c`. You now have to include these files in the compilation of your program.</p><p>You would use the following command for Go:</p><p>``` are <del>lang=go </del>rule=package.xml <del>list-access </del>name-access . ```</p><p>and it would generate in `man/man.go` and `help/help.go` the Go source files.</p><p>You would use the following command for Ada:</p><p>``` are <del>lang=Ada -o src </del>rule=package.xml <del>list-access </del>name-access . ```</p><p>and it would generate `src/man.ads`, `src/man.adb`, `src/help.ads` and `src/help.adb`.</p><ol><li><ol><li>Using the resource</li></ol></li></ol><p>The code generator emits by default a set of type and function declaration that give access to the resource. For C, the following structure and declaration are generated in the header for the `man` resource:</p><p>```C struct man_content {</p><pre><code> const unsigned char *content;
size_t size;
time_t modtime;
int format;
</code></pre><p>};</p><p>extern const struct man_content* man_get_content(const char* name); ```</p><p>But for the Go language, the `man` generated package declares:</p><p>```Go type Content struct {</p><pre><code> Content []byte
Size int64
Modtime int64
Format int
</code></pre><p>} ... func Get_content(name string) (*Content) {...} ```</p><p>And for Ada, it will generate:</p><p>```Ada package Resources.Man is</p><pre><code> type Content_Access is access constant Ada.Streams.Stream_Element_Array;
type Name_Access is access constant String;
type Format_Type is (FILE_RAW, FILE_GZIP);
type Content_Type is record
Name : Name_Access;
Content : Content_Access;
Modtime : Interfaces.C.long := 0;
Format : Format_Type := FILE_RAW;
end record;
Null_Content : constant Content_Type;
</code></pre><pre><code> type Name_Array is array (Natural range <>) of Name_Access;
Names : constant Name_Array;
</code></pre><pre><code> function Get_Content (Name : String) return Content_Type;
</code></pre><p>private</p><pre><code> ...
</code></pre><p>end Resources.Man; ```</p><p>You can avoid the type declaration by using the `--no-type-declaration` option and in that case you have to make available these types somehow. In Ada, this can easily be made by providing these types in a parent Ada package (see the <a href="Embedding help and documentation in Ada">Embedding help and documentation in Ada</a>(https://gitlab.com/stcarrez/resource-embedder/tree/master/examples/ada-help) for the example).</p><p>Now with the generated code, you only need to call the generated get content method to obtain the embedded content. Simple!</p><ol><li><ol><li>Examples</li></ol></li></ol><p>The project proposes several detailed examples to illustrate various integrations.</p><p>This first set of example shows how to you can embed configuration files in a C, Ada or Go program. The Advance Resource Embedder simply puts the configuration files in an array of bytes that can easily be retrieved by a generated function.</p><ul><li><a href="Embedding configuration files in C">Embedding configuration files in C</a>(https://gitlab.com/stcarrez/resource-embedder/tree/master/examples/c-config)</li><li><a href="Embedding configuration files in Ada">Embedding configuration files in Ada</a>(https://gitlab.com/stcarrez/resource-embedder/tree/master/examples/ada-config)</li><li><a href="Embedding configuration files in Go">Embedding configuration files in Go</a>(https://gitlab.com/stcarrez/resource-embedder/tree/master/examples/go-config)</li></ul><p>A second set of example is more advanced by the use of an XML file that describes what must be embedded with the transformations that must be made. It creates two distinct resource sets `help` and `man`. The `help` resource set is composed of a set of fixed documentation files provided in the example. The `man` resource set is created by running the `man` Unix command on various names to embed the man page of `ls`, `pwd` and `sh`.</p><ul><li><a href="Embedding help and documentation in C">Embedding help and documentation in C</a>(https://gitlab.com/stcarrez/resource-embedder/tree/master/examples/c-help)</li><li><a href="Embedding help and documentation in Ada">Embedding help and documentation in Ada</a>(https://gitlab.com/stcarrez/resource-embedder/tree/master/examples/ada-help)</li><li><a href="Embedding help and documentation in Go">Embedding help and documentation in Go</a>(https://gitlab.com/stcarrez/resource-embedder/tree/master/examples/go-help)</li></ul><p>More specific examples show how to make specific transformations on the files before integrating them:</p><ul><li><a href="Embedding web files in C">Embedding web files in C</a>(https://gitlab.com/stcarrez/resource-embedder/tree/master/examples/c-web)</li><li><a href="Embedding merged properties in Ada">Embedding merged properties in Ada</a>(https://gitlab.com/stcarrez/resource-embedder/tree/master/examples/ada-bundles)</li></ul><ol><li><ol><li>License considerations</li></ol></li></ol><p>The generated code produced by <a href="Advance Resource Embedder">Advance Resource Embedder</a>(https://gitlab.com/stcarrez/resource-embedder) is not covered by any license. However, when you integrate some resource files with the tool, the generated code will contain the original file in some binary form and therefore it may be covered by the license associated with these resource files.</p><p>For example, if you include a file covered by the GNU General Public license, then the generated file is covered by the GPL.</p><ol><li><ol><li>How can you get Advanced Resource Embedder?</li></ol></li></ol><ol><li><ol><li><ol><li>Use the source Luke!</li></ol></li></ol></li></ol><p>Quick instructions to build the tool:</p><p>``` git clone --recursive <a href="https://gitlab.com/stcarrez/resource-embedder.git">https://gitlab.com/stcarrez/resource-embedder.git</a> cd resource-embedder make build test install ```</p><ol><li><ol><li><ol><li>Debian Packages for x86_64</li></ol></li></ol></li></ol><p>You can install ARE by using the Debian 10 and Ubuntu 20.04 or 18.04 packages. First, setup to accept the signed packages:</p><p>``` wget -O - <a href="https://apt.vacs.fr/apt.vacs.fr.gpg.key">https://apt.vacs.fr/apt.vacs.fr.gpg.key</a> | sudo apt-key add - ```</p><p>and choose one of the `echo` command according to your Linux distribution:</p><p>Ubuntu 20.04 ``` echo "deb <a href="https://apt.vacs.fr/ubuntu-focal">https://apt.vacs.fr/ubuntu-focal</a> focal main" | sudo tee -a /etc/apt/sources.list.d/vacs.list ```</p><p>Ubuntu 18.04 ``` echo "deb <a href="https://apt.vacs.fr/ubuntu-bionic">https://apt.vacs.fr/ubuntu-bionic</a> bionic main" | sudo tee -a /etc/apt/sources.list.d/vacs.list ```</p><p>Debian 10 ``` echo "deb <a href="https://apt.vacs.fr/debian-buster">https://apt.vacs.fr/debian-buster</a> buster main" | sudo tee -a /etc/apt/sources.list.d/vacs.list ```</p><p>Then, launch the apt update command:</p><p>``` sudo apt-get update ```</p><p>and install the tool using:</p><p>``` sudo apt-get install -y are ```</p><ol><li><ol><li>Conclusion</li></ol></li></ol><p>Embedding files, scripts, documentation and other contents in C and Ada is easily done by using the <a href="Advance Resource Embedder">Advance Resource Embedder</a>(https://gitlab.com/stcarrez/resource-embedder). Go developers are already familiar with these mechanisms thanks to the `go:embed` directive. But the tool provides custom transformations that gives more flexibility for the integration.</p><p>Some benefits in embedding contents in final binaries:</p><ul><li>the content is embedded within the readonly `.rodata` section,</li><li>you avoid at least 3 system calls to access the content: an `open`, a `read` and a `close`,</li><li>you reduce the size of your application on the disk: contents are contiguous within the `.rodata` section</li><pre><code> whereas written on a file system they each require full dedicated data blocks (4K in most cases),
</code></pre><li>you get a portable `mmap` of your files for free and without pain.</li></ul><p>If you want to know more about the tool, have a look at its documentation:</p><ul><li><a href="Resource Embedder Guide">Resource Embedder Guide</a>(https://resource-embedder.readthedocs.io/en/latest/) <a href="PDF">PDF</a>(https://gitlab.com/stcarrez/resource-embedder/blob/master/docs/are-book.pdf)</li><li>Man page: <a href="are (1)">are (1)</a>(https://gitlab.com/stcarrez/resource-embedder/blob/master/docs/are.md)</li></ul><p>and if you have ideas for improvements, fork the project and submit a pull request!</p></div> Generating a REST Ada client with OpenAPI and Swagger Codegenurn:md5:afc8aafa41a0e88a96fa5eb6068f34732017-10-08T18:32:00+00:002017-10-08T18:32:00+00:00Stephane CarrezAdageneratorRESTAPI
<div class="post-text"><p><div class="wiki-img-center"><div class="wiki-img-inner"><img src="/images/Ada/swagger-ada-generator.png" longdesc="OpenAPI and Swagger Codegen" alt="swagger-ada-generator.png"></img></div></div></p><h3>Writing an OpenAPI document</h3><p>The OpenAPI document is either a JSON or a YAML file that describes the REST API operations. The document can be used both for the documentation of the API and for the code generation in several programming language. We will see briefly through the Petstore example how the OpenAPI document is organized. The full OpenAPI document is available in <a href="https://github.com/stcarrez/swagger-ada-petstore/blob/master/petstore.yaml">petstore.yaml</a>.</p><h4>General description</h4><p>A first part of the OpenAPI document provides a general description of the API. This includes the general description, the terms of service, the license and some contact information.</p><pre><code class="lang-yaml">swagger: '2.0'
info:
description: 'This is a sample server Petstore server. You can find out more about Swagger at [http://swagger.io](http://s
wagger.io) or on [irc.freenode.net, #swagger](http://swagger.io/irc/). For this sample, you can use the api key `special-key
` to test the authorization filters.'
version: 1.0.0
title: Swagger Petstore
termsOfService: 'http://swagger.io/terms/'
contact:
email: apiteam@swagger.io
license:
name: Apache 2.0
url: 'http://www.apache.org/licenses/LICENSE-2.0.html'
host: petstore.swagger.io
basePath: /v2
</code></pre><h4>Type description</h4><p>The OpenAPI document can also describe types which are used by the REST operations. These types provide a description of how the data is organized and passed through the API operations.</p><p>It is possible to describe almost all possible types from simple properties, group of properties up to complex types including arrays. For example a <code>Pet</code> type is made of several properties each of them having a name, a type and other information to describe how the type is serialized.</p><pre><code class="lang-yaml">definitions:
Pet:
title: a Pet
description: A pet for sale in the pet store
type: object
required:
- name
- photoUrls
properties:
id:
type: integer
format: int64
category:
$ref: '#/definitions/Category'
name:
type: string
example: doggie
photoUrls:
type: array
xml:
name: photoUrl
wrapped: true
items:
type: string
tags:
type: array
xml:
name: tag
wrapped: true
items:
$ref: '#/definitions/Tag'
status:
type: string
description: pet status in the store
enum:
- available
- pending
- sold
xml:
name: Pet
</code></pre><p>In this example, the <code>Pet</code> type contains 6 properties (<code>id</code>, <code>category</code>, <code>name</code>, <code>photoUrls</code>, <code>tags</code>, <code>status</code>) and refers to two other types <code>Category</code> and <code>Tag</code>.</p><h4>Operation description</h4><p>Operations are introduced by the <code>paths</code> object in the OpenAPI document. This section describes the possible paths that can be used by URL and the associated operation. Some operations receive their parameter within the path and this is represented by the <code>{name}</code> notation.</p><p>The operation description indicates the HTTP method that is used <code>get</code>, <code>post</code>, <code>put</code> or <code>delete</code>.</p><p>The following definition describes the <code>getPetById</code> operation.</p><pre><code class="lang-yaml">paths:
'/pet/{petId}':
get:
tags:
- pet
summary: Find pet by ID
description: Returns a single pet
operationId: getPetById
produces:
- application/xml
- application/json
parameters:
- name: petId
in: path
description: ID of pet to return
required: true
type: integer
format: int64
responses:
'200':
description: successful operation
schema:
$ref: '#/definitions/Pet'
'400':
description: Invalid ID supplied
'404':
description: Pet not found
security:
- api_key: []
</code></pre><p>The <code>summary</code> and <code>description</code> are used for the documentation purposes. The <code>operationId</code> is used by code generators to provide an operation name that a target programming language can use. The <code>produces</code> section indicates the media types that are supported by the operation and which are generated for the response. The <code>parameters</code> section represents all the operation parameters. Some parameters can be extracted from the path (which is the case for the <code>petId</code> parameter) and some others can be passed as query parameter.</p><p>The <code>responses</code> section describes the possible responses for the operation as well as the format used by the response. In this example, the operation returns an object described by the <code>Pet</code> type.</p><h3>Using Swagger Codegen</h3><p>The documentation and the Ada client are generated from the OpenAPI document by using the <a href="https://github.com/swagger-api/swagger-codegen">Swagger Codegen</a> generator. The generator is a Java program that is packaged within a jar file. It must be launched by the Java 7 or Java 8 runtime.</p><h4>Generating the documentation</h4><p>The HTML documentation is generated from the OpenAPI document by using the following command:</p><pre><code> java -jar swagger-codegen-cli.jar generate -l html -i petstore.yaml -o doc
</code></pre><h4>Generating the Ada client</h4><p>To generate the Ada client, you will use the <code>-l ada</code> option to use the Ada code generator. The OpenAPI document is passed with the <code>-i</code> option.</p><pre><code> java -jar swagger-codegen-cli.jar generate -l ada -i petstore.yaml -o client \
-DprojectName=Petstore --model-package Samples.Petstore
</code></pre><p>The Ada generator uses two options to control the generation. The <code>-DprojectName=Petstore</code> option allows to control the name of the generated GNAT project and the <code>--model-package</code> option controls the name of the Ada package for the generated code.</p><p>The Ada generator will create the following Ada packages:</p><ul><li><code>Samples.Petstore.Models</code> is the package that contains all the types described in the OpenAPI document. Each OpenAPI type is represented by an Ada record and it</li></ul><p>is also completed by an instantiation of the <code>Ada.Containers.Vectors</code> package for the representation of arrays of the given type. The <code>Models</code> package also provides <code>Serialize</code> and <code>Deserialize</code> procedures for the serialization and deserialization of the data over JSON or XML streams.</p><ul><li><code>Samples.Petstore.Clients</code> is the package that declares the <code>Client_Type</code> tagged record which provides all the operations for the OpenAPI document.</li></ul><p>For the <code>Pet</code> type describe previously, the Ada generator produces the following code extract:</p><pre><code class="lang-ada">package Samples.Petstore.Models is
...
type Pet_Type is
record
Id : Swagger.Long;
Category : Samples.Petstore.Models.Category_Type;
Name : Swagger.UString;
Photo_Urls : Swagger.UString_Vectors.Vector;
Tags : Samples.Petstore.Models.Tag_Type_Vectors.Vector;
Status : Swagger.UString;
end record;
...
end Samples.Petstore.Models;
</code></pre><p>and for the operation it generates the following code:</p><pre><code class="lang-ada">package Samples.Petstore.Clients is
...
type Client_Type is new Swagger.Clients.Client_Type with null record;
procedure Get_Pet_By_Id
(Client : in out Client_Type;
Pet_Id : in Swagger.Long;
Result : out Samples.Petstore.Models.Pet_Type);
...
end Samples.Petstore.Clients;
</code></pre><h3>Using the REST Ada client</h3><h4>Initialization</h4><p>The HTTP/REST support is provided by <a href="https://github.com/stcarrez/ada-util">Ada Util</a> and encapsulated by <a href="https://github.com/stcarrez/swagger-ada">Swagger Ada</a>. The <a href="https://github.com/stcarrez/ada-util">Ada Util</a> library also takes care of the JSON and XML serialization and deserialization. If you want to use Curl, you should initialize with the following:</p><pre><code class="lang-ada">with Util.Http.Clients.Curl;
...
Util.Http.Clients.Curl.Register;
</code></pre><p>But if you want to use <a href="http://libre.adacore.com/libre/tools/aws/">AWS</a>, you will initialize with:</p><pre><code class="lang-ada">with Util.Http.Clients.Web;
...
Util.Http.Clients.Web.Register;
</code></pre><p>After the initialization is done, you will declare a client instance to access the API operations:</p><pre><code class="lang-ada">with Samples.Petstore.Clients;
...
C : Samples.Petstore.Clients.Client_Type;
</code></pre><p>And you should initialize the server base URL you want to connect to. To use the live <a href="http://petstore.swagger.io/">Swagger Petstore</a> service you can set the server base URL as follows:</p><pre><code class="lang-ada"> C.Set_Server ("http://petstore.swagger.io/v2");
</code></pre><p>At this stage, you can use the generated operation by calling operations on the client.</p><h4>Calling a REST operation</h4><p>Let's retrieve some pet information by calling the <code>Get_Pet_By_Id</code> operation described previously. This operation needs an integer as input parameter and returns a <code>Pet_Type</code> object that contains all the pet information. You will first declare the pet instance as follows:</p><pre><code class="lang-ada">with Samples.Petstore.Models;
...
Pet : Samples.Petstore.Models.Pet_Type;
</code></pre><p>And then call the <code>Get_Pet_By_Id</code> operation:</p><pre><code class="lang-ada"> C.Get_Pet_By_Id (768, Pet);
</code></pre><p>At this stage, you can access information from the <code>Pet</code> instance:</p><pre><code class="lang-ada">with Ada.Text_IO;
...
Ada.Text_IO.Put_Line ("Id : " & Swagger.Long'Image (Pet.Id));
Ada.Text_IO.Put_Line ("Name : " & Swagger.To_String (Pet.Name));
Ada.Text_IO.Put_Line ("Status : " & Swagger.To_String (Pet.Status));
</code></pre><p>The <a href="https://github.com/stcarrez/swagger-ada-petstore">Swagger Ada Petstore</a> illustrates other uses of the generated operations. It allows to list the inventory, list the pets with a given status, add a pet and so on...</p><h3>Conclusion and references</h3><p>The <a href="https://github.com/OAI/OpenAPI-Specification">OpenAPI Specification</a> provides a standard way to describe REST operations. The <a href="https://github.com/swagger-api/swagger-codegen">Swagger Codegen</a> is the generator to be used to simplify the implementation of REST clients in many programming languages and to generate the documentation of the API. The Ada code generator only supports the client side but the server code generation is under work.</p><p>The sources of the petstore samples are available:</p><ul><li><a href="https://github.com/stcarrez/swagger-ada-petstore/blob/master/src/petstore.adb">petstore.adb</a></li><li><a href="https://github.com/stcarrez/swagger-ada-petstore/blob/master/src/client/samples-petstore-models.ads">samples-petstore-models.ads</a></li><li><a href="https://github.com/stcarrez/swagger-ada-petstore/blob/master/src/client/samples-petstore-models.adb">samples-petstore-models.adb</a></li><li><a href="https://github.com/stcarrez/swagger-ada-petstore/blob/master/src/client/samples-petstore-clients.ads">samples-petstore-clients.ads</a></li><li><a href="https://github.com/stcarrez/swagger-ada-petstore/blob/master/src/client/samples-petstore-clients.adb">samples-petstore-clients.adb</a></li></ul><p>The <a href="https://apis.guru/openapi-directory/">APIs.guru</a> lists more than 550 API descriptions from various providers such as Amazon, Google, Microsoft and many other online services. They are now available to the Ada community!</p></div> Dynamo 0.7.0 is availableurn:md5:0c85b61b2cf36654e04ed478a2e7504f2014-05-12T19:28:07+00:002014-05-12T19:28:07+00:00Stephane CarrezreleaseAdaMDEUMLgenerator
<div class="post-text"><p><a href="https://code.google.com/p/ada-gen/">Dynamo</a> is a code generator used to generate <a href="https://code.google.com/p/ada-gen/">Ada Web Application</a> or database mappings.</p><ul><li>New project template to generate Gtk Ada application</li><li>Register the new module in the application when they are added</li><li>Update the current testsuite when new tests are added</li><li>New stereotype for Ada bean generation</li><li>Support for the creation of Debian packages</li><li>New command <code>add-form</code> and <code>add-module-operation</code></li></ul><p>You can download the new version at <a href="http://download.vacs.fr/dynamo/dynamo-0.7.0.tar.gz">http://download.vacs.fr/dynamo/dynamo-0.7.0.tar.gz</a></p></div> Dynamo 0.6.0 is availableurn:md5:e17c073fa27df4c54e7f847105f8279a2013-02-10T22:15:26+00:002013-02-10T22:15:26+00:00Stephane CarrezAdaMDEUMLgeneratorrelease
<div class="post-text"><p>Dynamo is a tool to help developers write some types of Ada Applications which use the <a href="http://code.google.com/p/ada-asf">Ada Server Faces</a> or <a href="http://code.google.com/p/ada-ado">Ada Database Objects</a> frameworks. Dynamo provides several commands to perform one specific task in the development process: creation of an application, generation of database model, generation of Ada model, creation of database.</p><p>The new version of Dynamo provides:</p><ul><li>A new command <code>build-doc</code> to extract some documentation from the sources,</li><li>The generation of MySQL and SQLite schemas from UML models,</li><li>The generation of Ada database mappings from UML models,</li><li>The generation of Ada beans from the UML models,</li><li>A new project template for command line tools using ADO,</li><li>A new distribution command to merge the resource bundles.</li></ul><p>The most important feature is probably the Ada code generation from a UML class diagram. With this, you can design the data model of an application using <a href="http://argouml.tigris.org/">ArgoUML</a> and generate the Ada model files that will be used to access the database easily through the <a href="http://code.google.com/p/ada-ado">Ada Database Objects</a> library. The tool will also generate the SQL database schema so that everything is concistent from your UML model, to the Ada implementation and the database tables.</p><p>The short tutorial below indicates how to design a UML model with <a href="http://argouml.tigris.org/">ArgoUML</a>, generate the Ada model files, the SQL files and create the MySQL database.</p><div style='width: 100%; float: left;'>
<iframe width="640" height="480" src="http://www.youtube.com/embed/5q542_gVDd0" frameborder="0" allowfullscreen></iframe>
</div>
<p>The Dynamo tool is available at <a href="http://code.google.com/p/ada-gen">http://code.google.com/p/ada-gen</a>.</p><p>To build Dynamo, you will need:</p><ul><li><a href="http://code.google.com/p/ada-asf/downloads/list">Ada ASF 0.5.0</a></li><li><a href="http://code.google.com/p/ada-ado/downloads/list">Ada ADO 0.4.0</a></li><li><a href="http://code.google.com/p/ada-el/downloads/list">Ada EL 1.4.2</a></li><li><a href="http://code.google.com/p/ada-util/downloads/list">Ada Util 1.6.1</a></li><li><a href="http://libre.adacore.com/libre/tools/xmlada/">XML Ada 4.x or 3.x</a></li></ul></div> Dynamo 0.5.0 is availableurn:md5:8c146a2482ecd8f9a4c9892dc78238cb2012-05-20T20:40:05+00:002012-05-20T20:40:05+00:00Stephane CarrezAdageneratorrelease
<div class="post-text"><p>Dynamo is a tool to help developers write an Ada Web Application using the <a href="http://code.google.com/p/ada-asf">Ada Server Faces</a> and the <a href="http://code.google.com/p/ada-ado">Ada Database Objects</a> frameworks. Dynamo provides several commands to perform one specific task in the development process: creation of an application, generation of database model, generation of Ada model, creation of database.</p><p>The new version of Dynamo provides:</p><ul><li>Support multi-line comments in XML mappings,</li><li>Generate List_Bean types for the XML mapped queries,</li><li>Add support for Ada enum generation,</li><li>Add test template generation,</li><li>Add AWA service template generation,</li><li>Add support for blob model mapping,</li><li>New command 'add-ajax-form', 'add-query', 'dist', 'create-plugin'</li></ul><p>The Dynamo tool is available at <a href="http://code.google.com/p/ada-gen">http://code.google.com/p/ada-gen</a>. To build Dynamo, you will need:</p><ul><li><a href="http://code.google.com/p/ada-asf/downloads/list">Ada ASF 0.4.0</a></li><li><a href="http://code.google.com/p/ada-ado/downloads/list">Ada ADO 0.3.0</a></li><li><a href="http://code.google.com/p/ada-el/downloads/list">Ada EL 1.4.0</a></li><li><a href="http://libre.adacore.com/libre/tools/xmlada/">XML Ada 4.x or 3.x</a></li></ul></div> Ada perfect hash generation with gperfhashurn:md5:7ad85c1533dcc1b5e103c64539e26bf72012-01-17T21:53:00+00:002012-01-17T21:53:00+00:00Stephane CarrezAdageneratorTutorial
<div class="post-text"><p>A <a href="https://en.wikipedia.org/wiki/Perfect_hash_function">perfect hash function</a> is a function that returns a distinct hash number for each keyword of a well defined set. <a href="https://www.gnu.org/software/gperf/">gperf</a> is famous and well known perfect hash generator used for C or C++ languages. Ada is not supported.</p><p>The <a href="https://github.com/stcarrez/ada-util/blob/master/samples/gperfhash.adb">gperfhash</a> is a sample from the <a href="https://github.com/stcarrez/ada-util">Ada Utility Library</a> which generates an Ada package that implements such perfect hash operation. It is not as complete as gperf but allows to easily get a hash operation. The gperfhash tool uses the GNAT package <code>GNAT.Perfect_Hash_Generators</code>.</p><h4>Pre requisite</h4><p>Since the gperfhash tool is provided by the Ada Util samples, you must build these samples with the following command:</p><pre><code>$ gnatmake -Psamples
</code></pre><h4>Define a keyword file</h4><p>First, create a file which contains one keyword on each line. For example, let's write a <code>keywords.txt</code> file which contains the following three keywords:</p><pre><code>int
select
print
</code></pre><h4>Generate the package</h4><p>Run the gperfhash tool and give it the package name.</p><pre><code>$ gperfhash -p Hashing keywords.txt
</code></pre><p>The package defines a <code>Hash</code> and an <code>Is_Keyword</code> function. The Hash function returns a hash number for each string passed as argument. The hash number will be different for each string that matches one of our keyword. You can give a string not in the keyword list, in that case the hash function will return a number that collides with a hash number of one or our keyword.</p><p>The <code>Is_Keyword</code> function allows to check whether a string is a keyword of the list. This is very useful when you just want to know whether a string is a reserved keyword in some application.</p><p>The package specification is the following:</p><pre><code class="lang-ada">-- Generated by gperfhash
package Hashing is
function Hash (S : String) return Natural;
-- Returns true if the string <b>S</b> is a keyword.
function Is_Keyword (S : in String) return Boolean;
type Name_Access is access constant String;
type Keyword_Array is array (Natural range <>) of Name_Access;
Keywords : constant Keyword_Array;
private
...
end Hashing;
</code></pre><h4>How to use the hash</h4><p>Using the perfect hash generator is simple:</p><pre><code class="lang-ada">with Hashing;
if Hashing.Is_Keyword (S) then
-- 'S' is one of our keyword
else
-- No, it's not a keyword
end if;
</code></pre></div>