Incorporating files in a binary program can sometimes be a challenge. The Advance Resource Embedder is a flexible tool that collects files such as documentation, images, scripts, configuration files and generates a source code that contains these files.

The tool is able to apply some transformations on the collected files:

• it can run a Javascript minifier such as closure,
• it can compress CSS files by running yui-compressor,
• it can compress files by running gzip or another compression tool.

Once these transformations are executed, it invokes a target generator to produce a source file either in C, Ada or Go language. The generated source file can then be used in the final program and taken into account during the compilation process of that program. At the end, the binary will contain the embedded files with their optional transformations.

The process to use ARE is simple:

• You describe the resources that you want to embed. The description is either made on command line arguments or by writing an XML file. The XML description gives more flexibility as it allows to define a transformation rule that must be executed on the original file before being embedded. This allows to minify a Javascript or CSS file, compress some files and even encrypt a file before its integration.
• You run the ARE command with your target language and rule description and you give the tool a list of directories that must be scanned to identify the files that must be collected. The ARE tool scan the directories according to the patterns that you have given either on the command line or in the XML rule description. After identifying the files, the tool applies the rules and execute the transformations. The ARE tool then invokes the target language generator that writes one or several files depending on the list of resources.
• Once the files are generated, you use them in your program and add them in your build process as they are now part of your sources. After building your program, it now embeds the resource files that were collected and optionally transformed.

Defining resources to embed

The first step is to write some package.xml file which describes a list of resources with their content. The root XML element is package and each resource is described by a resource XML element. The resource is assigned a name that will be used by the code generator. The C generator will use the name as a prefix for the C function and C type declaration. The Ada and Go generator will use that name for the Ada or Go package.

The following resource definition declares the Help resource. It contains an installation rule that will copy the files under the help directory in the resource set. Only files matching the .txt pattern will be taken into account.

<package>
  <resource name='Help'>
    <install mode='copy'>
      <fileset dir="help">
        <include name="**/*.txt"/>
      </fileset>
    </install>
  </resource>
  ...
</package>

The next resource definition will run an external program to get the content that must be embedded. The man directory is scanned and it will execute the command man #{name} on each filename found. That directory contains the empty files ls, pwd and sh and this will run and embed the man page for these commands.

<package>
   ...
  <resource name='Man'>
    <install mode='exec'>
      <command output='#{dst}'>man #{name}</command>
      <fileset dir="man">
        <include name="*"/>
      </fileset>
    </install>
  </resource>
</package>

You may run other commands such as:

<command>closure --charset UTF-8 #{src} --js_output_file #{dst}</command>
<command>yui-compressor --type css --charset UTF-8 -o #{dst} #{src}</command>
<command output="#{dst}">gzip --no-name -c #{src}</command>

Running the Advanced Resource Embedder

The tool has several options that allows you to control the target code generator and tune the generation according to your needs. Assuming that the files to embed are located in the current directory, you would use the following command to generate C source files in the src directory:

are --lang=c -o src --rule=package.xml --list-access --name-access  .

For C, this would generate a src/man.h, src/man.c, src/help.h and src/help.c. You now have to include these files in the compilation of your program.

You would use the following command for Go:

are --lang=go --rule=package.xml --list-access --name-access  .

and it would generate in man/man.go and help/help.go the Go source files.

You would use the following command for Ada:

are --lang=Ada -o src --rule=package.xml --list-access --name-access  .

and it would generate src/man.ads, src/man.adb, src/help.ads and src/help.adb.

Using the resource

The code generator emits by default a set of type and function declaration that give access to the resource. For C, the following structure and declaration are generated in the header for the man resource:

struct man_content {
  const unsigned char *content;
  size_t size;
  time_t modtime;
  int format;
};

extern const struct man_content* man_get_content(const char* name);

But for the Go language, the man generated package declares:

type Content struct {
    Content  []byte
    Size  int64
    Modtime  int64
    Format   int
}
...
func Get_content(name string) (*Content) {...}

And for Ada, it will generate:

package Resources.Man is
   type Content_Access is access constant Ada.Streams.Stream_Element_Array;
   type Name_Access is access constant String;
   type Format_Type is (FILE_RAW, FILE_GZIP);
   type Content_Type is record
      Name    : Name_Access;
      Content : Content_Access;
      Modtime : Interfaces.C.long := 0;
      Format  : Format_Type := FILE_RAW;
   end record;
   Null_Content : constant Content_Type;

   type Name_Array is array (Natural range <>) of Name_Access;
   Names : constant Name_Array;

   function Get_Content (Name : String) return Content_Type;
private
   ...
end Resources.Man;

You can avoid the type declaration by using the --no-type-declaration option and in that case you have to make available these types somehow. In Ada, this can easily be made by providing these types in a parent Ada package (see the Embedding help and documentation in Ada for the example).

Now with the generated code, you only need to call the generated get content method to obtain the embedded content. Simple!

Examples

The project proposes several detailed examples to illustrate various integrations.

This first set of example shows how to you can embed configuration files in a C, Ada or Go program. The Advance Resource Embedder simply puts the configuration files in an array of bytes that can easily be retrieved by a generated function.

• Embedding configuration files in C
• Embedding configuration files in Ada
• Embedding configuration files in Go

A second set of example is more advanced by the use of an XML file that describes what must be embedded with the transformations that must be made. It creates two distinct resource sets help and man. The help resource set is composed of a set of fixed documentation files provided in the example. The man resource set is created by running the man Unix command on various names to embed the man page of ls, pwd and sh.

• Embedding help and documentation in C
• Embedding help and documentation in Ada
• Embedding help and documentation in Go

More specific examples show how to make specific transformations on the files before integrating them:

• Embedding web files in C
• Embedding merged properties in Ada

License considerations

The generated code produced by Advance Resource Embedder is not covered by any license. However, when you integrate some resource files with the tool, the generated code will contain the original file in some binary form and therefore it may be covered by the license associated with these resource files.

For example, if you include a file covered by the GNU General Public license, then the generated file is covered by the GPL.

How can you get Advanced Resource Embedder?

Use the source Luke!

Quick instructions to build the tool:

git clone --recursive https://gitlab.com/stcarrez/resource-embedder.git
cd resource-embedder
make build test install

Debian Packages for x86_64

You can install ARE by using the Debian 10 and Ubuntu 20.04 or 18.04 packages. First, setup to accept the signed packages:

wget -O - https://apt.vacs.fr/apt.vacs.fr.gpg.key | sudo apt-key add -

and choose one of the echo command according to your Linux distribution:

Ubuntu 20.04

echo "deb https://apt.vacs.fr/ubuntu-focal focal main" | sudo tee -a /etc/apt/sources.list.d/vacs.list

Ubuntu 18.04

echo "deb https://apt.vacs.fr/ubuntu-bionic bionic main" | sudo tee -a /etc/apt/sources.list.d/vacs.list

Debian 10

echo "deb https://apt.vacs.fr/debian-buster buster main" | sudo tee -a /etc/apt/sources.list.d/vacs.list

Then, launch the apt update command:

sudo apt-get update

and install the tool using:

sudo apt-get install -y are

Conclusion

Embedding files, scripts, documentation and other contents in C and Ada is easily done by using the Advance Resource Embedder. Go developers are already familiar with these mechanisms thanks to the go:embed directive. But the tool provides custom transformations that gives more flexibility for the integration.

Some benefits in embedding contents in final binaries:

• the content is embedded within the readonly .rodata section,
• you avoid at least 3 system calls to access the content: an open, a read and a close,
• you reduce the size of your application on the disk: contents are contiguous within the .rodata section whereas written on a file system they each require full dedicated data blocks (4K in most cases),
• you get a portable mmap of your files for free and without pain.

If you want to know more about the tool, have a look at its documentation:

• Resource Embedder Guide PDF
• Man page: are (1)

and if you have ideas for improvements, fork the project and submit a pull request!