IO stream composition is a powerful mechanism that has been provided by Java platform since its early days. It allows one or more transformations to be performed when reading or writing some content. Doing such transformation is transparent to the reader or the writer. For example, in the reading process, it allows to first decode the content in Base64, then decrypt the binary stream and then decompress that decrypted binary stream, all this in a transparent manner. The writing process would first compress what is written, then encrypt and encode in Base64. All these transformations are activated by connecting one stream object to another. In Ada, such IO stream composition is possible with Ada Utility Library.

To be able to provide this IO stream combination, the Ada Utility Library defines two Ada types: the Input_Stream and the Output_Stream limited interfaces. The Input_Stream interface only defines a Read procedure and the Output_Stream interface defines the Write, Flush and Close procedures. By implementing these interfaces, it is possible to provide stream objects that can be combined together.

The Ada Utility Library provides stream types that implement these interfaces so that it is possible to read or write files, sockets and system pipes. In many cases, the concrete type implements both interfaces so that reading or writing is possible. This is the case for File_Stream which allows to read or write on a file, the Socket_Stream which handles sockets by using GNAT sockets. The Pipe_Stream on its side allows to launch an external program and either read its output, available through the Input_Stream, or feed the external program with some input by using the Output_Stream.

The Ada Utility Library also provides stream objects that make transformation on the data through various data encoders. The Ada library supports the following encoders:

• Base 16, Base 64,
• AES encryption or decryption,
• LZMA compression or decompression

Other encoders could be added and it is always possible to provide custom transformations by implementing the Input_Stream and Output_Stream interfaces.

The last part that completes the IO stream framework is the serialization framework. That framework defines and provides interface and types to read or write a CSV, XML, JSON or HTTP form stream. The serialization framework uses either the Input_Stream or the Output_Stream interfaces to either read or write the content. The serialization framework defines operations in a way that allows to read or write these streams independently of their representation format.

LZMA Compression

Let's have a look at compressing a file by using the Util.Streams framework. First we need a File_Stream that is configured to read the file to compress and we need another File_Stream configured for writing to save in another file. The first file is opened by using the Open procedure and the In_File mode while the second one is using Create and the Out_File mode. The File_Stream is using the Ada Stream_IO standard package to access files.

with Util.Streams.Files;

   In_Stream  : aliased Util.Streams.Files.File_Stream;
   Out_Stream : aliased Util.Streams.Files.File_Stream;
   
   In_Stream.Open (Mode => Ada.Streams.Stream_IO.In_File, Name => Source);
   Out_Stream.Create (Mode => Ada.Streams.Stream_IO.Out_File, Name => Destination);

In the middle of these two streams, we are going to use a Compress_Stream whose job is to compress the data and write the compressed result to a target stream. The compression stream is configured by using the Initialize procedure and it is configured to write on the Out_Stream file stream. The compression stream needs a buffer and its size is configured with the Size parameter.

with Util.Streams.Lzma;

   Compressor : aliased Util.Streams.Lzma.Compress_Stream;
   
   Compressor.Initialize (Output => Out_Stream'Unchecked_Access, Size => 32768);

To feed the compressor stream with the input file, we are going to use the Copy procedure. This procedure reads the content from the In_Stream and writes what is read to the Compressor stream.

   Util.Streams.Copy (From => In_Stream, Into => Compressor);

Flushing and closing the files is automatically handled by a Finalize procedure on the File_Stream type.

Complete source example: compress.adb

LZMA Decompression

The LZMA decompression is very close to the LZMA compression but instead it uses the Decompress_Stream. The complete decompression method is the following:

procedure Decompress_File (Source      : in String;
                           Destination : in String) is
   In_Stream    : aliased Util.Streams.Files.File_Stream;
   Out_Stream   : aliased Util.Streams.Files.File_Stream;
   Decompressor : aliased Util.Streams.Lzma.Decompress_Stream;
begin
   In_Stream.Open (Mode => Ada.Streams.Stream_IO.In_File, Name => Source);
   Out_Stream.Create (Mode => Ada.Streams.Stream_IO.Out_File, Name => Destination);
   Decompressor.Initialize (Input => In_Stream'Unchecked_Access, Size => 32768);
   Util.Streams.Copy (From => Decompressor, Into => Out_Stream);
end Decompress_File;

Complete source example: decompress.adb

AES Encryption

Encryption is a little bit more complex due to the encryption key that must be configured. The encryption is provided by the Encoding_Stream and it uses a Secret_Key to configure the encryption key. The Secret_Key is a limited type and it cannot be copied. To build the encryption key, one method consists in using the PBKDF2 algorithm described in RFC 8018. The user password is passed to the PBKDF2 algorithm configured to use the HMAC-256 hashing. The hash method is called on itself 20000 times in this example to produce the final encryption key.

with Util.Streams.AES;
with Util.Encoders.AES;
with Util.Encoders.KDF.PBKDF2_HMAC_SHA256;

   Cipher       : aliased Util.Streams.AES.Encoding_Stream;
   Password_Key : constant Util.Encoders.Secret_Key := Util.Encoders.Create (Password);
   Salt         : constant Util.Encoders.Secret_Key := Util.Encoders.Create ("fake-salt");
   Key          : Util.Encoders.Secret_Key (Length => Util.Encoders.AES.AES_256_Length);
   ...
      PBKDF2_HMAC_SHA256 (Password => Password_Key,
                          Salt     => Salt,
                          Counter  => 20000,
                          Result   => Key);

The encoding stream is able to produce or consume another stream. For the encryption, we are going to use the first mode and use the Produces procedure to configure the encryption to write on the Out_Stream file. Once configured, the Set_Key procedure must be called with the encryption key and the encryption method. The initial encryption IV vector can be configured by using the Set_IV procedure (not used by the example). As soon as the encryption key is configured, the encryption can start and the Cipher encoding stream can be used as an Output_Stream: we can write on it and it will encrypt the content before passing the result to the next stream. This means that we can use the same Copy procedure to read the input file and pass it through the encryption encoder.

   Cipher.Produces (Output => Out_Stream'Unchecked_Access, Size => 32768);
   Cipher.Set_Key (Secret => Key, Mode => Util.Encoders.AES.ECB);
   Util.Streams.Copy (From => In_Stream, Into => Cipher);

Complete source example: encrypt.adb

AES Decryption

Decryption is similar but it uses the Decoding_Stream type. Below is the complete example to decrypt the file:

procedure Decrypt_File (Source      : in String;
                        Destination : in String;
                        Password    : in String) is
   In_Stream    : aliased Util.Streams.Files.File_Stream;
   Out_Stream   : aliased Util.Streams.Files.File_Stream;
   Decipher     : aliased Util.Streams.AES.Decoding_Stream;
   Password_Key : constant Util.Encoders.Secret_Key := Util.Encoders.Create (Password);
   Salt         : constant Util.Encoders.Secret_Key := Util.Encoders.Create ("fake-salt");
   Key          : Util.Encoders.Secret_Key (Length => Util.Encoders.AES.AES_256_Length);
begin
   --  Generate a derived key from the password.
   PBKDF2_HMAC_SHA256 (Password => Password_Key,
                       Salt     => Salt,
                       Counter  => 20000,
                       Result   => Key);
                       
   --  Setup file -> input and cipher -> output file streams.
   In_Stream.Open (Ada.Streams.Stream_IO.In_File, Source);
   Out_Stream.Create (Mode => Ada.Streams.Stream_IO.Out_File, Name => Destination);
   Decipher.Produces (Output => Out_Stream'Access, Size => 32768);
   Decipher.Set_Key (Secret => Key, Mode => Util.Encoders.AES.ECB);
   
   --  Copy input to output through the cipher.
   Util.Streams.Copy (From => In_Stream, Into => Decipher);
end Decrypt_File;

Complete source example: decrypt.adb

Stream composition: LZMA > AES

Now, if we want to compress the stream before encryption, we can do this by connecting the Compressor to the Cipher stream and we only have to use the Compressor instead of the Cipher in the call to Copy.

   In_Stream.Open (Ada.Streams.Stream_IO.In_File, Source);
   Out_Stream.Create (Mode => Ada.Streams.Stream_IO.Out_File, Name => Destination);
   Cipher.Produces (Output => Out_Stream'Unchecked_Access, Size => 32768);
   Cipher.Set_Key (Secret => Key, Mode => Util.Encoders.AES.ECB);
   Compressor.Initialize (Output => Cipher'Unchecked_Access, Size => 4096);
   
   Util.Streams.Copy (From => In_Stream, Into => Compressor);

When Copy is called, the following will happen:

• first, it reads the In_Stream source file,
• the data is written to the Compress stream,
• the Compressor stream runs the LZMA compression and writes on the Cipher stream,
• the Cipher stream encrypts the data and writes on the Out_Stream,
• the Out_Stream writes on the destination file.

Complete source example: lzma_encrypt.adb

More stream composition: LZMA > AES > Base64

We can easily change the stream composition to encode in Base64 after the encryption. We only have to declare an instance of the Base64 Encoding_Stream and configure the encryption stream to write on the Base64 stream instead of the output file. The Base64 stream is configured to write on the output stream.

In_Stream    : aliased Util.Streams.Files.File_Stream;
Out_Stream   : aliased Util.Streams.Files.File_Stream;
Base64       : aliased Util.Streams.Base64.Encoding_Stream;
Cipher       : aliased Util.Streams.AES.Encoding_Stream;
Compressor   : aliased Util.Streams.Lzma.Compress_Stream;

   In_Stream.Open (Ada.Streams.Stream_IO.In_File, Source);
   Out_Stream.Create (Mode => Ada.Streams.Stream_IO.Out_File, Name => Destination);
   Base64.Produces (Output => Out_Stream'Unchecked_Access, Size => 32768);
   Cipher.Produces (Output => Base64'Unchecked_Access, Size => 32768);
   Cipher.Set_Key (Secret => Key, Mode => Util.Encoders.AES.ECB);
   Compressor.Initialize (Output => Cipher'Unchecked_Access, Size => 4096);

Complete source example: lzma_encrypt_b64.adb

Serialization

Serialization is achieved by using the Util.Serialize.IO packages and child packages and their specific types. The parent package defines the limited Output_Stream interface which inherit from the Util.Streams.Output_Stream interface. This allows to define specific operations to write various Ada types but also it provides common set of abstractions that allow to write either a JSON, XML, CSV and FORM (x-www-form-urlencoded) formats.

The target format is supported by a child package so that you only have to use the Output_Stream type declared in one of the JSON, XML, CSV or Form child package and use it transparently. There are some constraint if you want to switch from one output format to another while keeping the same code. These constraints comes from the nature of the different formats: XML has a notion of entity and attribute but other formats don't differentiate entities from attributes.

• A Start_Document procedure must be called first. Not all serialization method need it but it is required for JSON to produce a correct output.
• A Write_Entity procedure writes an XML entity of the given name. When used in JSON, it writes a JSON attribute.
• A Start_Entity procedure prepares the start of an XML entity or a JSON structure with a given name.
• A Write_Attribute procedure writes an XML attribute after a Start_Entity. When used in JSON, it writes a JSON attribute.
• A End_Entity procedure terminates an XML entity or a JSON structure that was opened by Start_Entity.
• At the end, the End_Document procedure must be called to finish correctly the output and terminate the JSON or XML content.

procedure Write (Stream : in out Util.Serialize.IO.Output_Stream'Class) is
begin
   Stream.Start_Document;
   Stream.Start_Entity ("person");
   Stream.Write_Entity ("name", "Harry Potter");
   Stream.Write_Entity ("gender", "male");
   Stream.Write_Entity ("age", 17);
   Stream.End_Entity ("person");
   Stream.End_Document;
end Write;

JSON Serialization

With the above Write procedure, if we want to produce a JSON stream, we only have to setup a JSON serializer. The JSON serializer is connected to a Print_Stream which provides a buffer and helper operations to write some text content. An instance of the Print_Stream is declared in Output and configured with a buffer size. The JSON serializer is then connected to it by calling the Initialize procedure and giving the Output parameter.

After writing the content, the JSON is stored in the Output print stream and it can be retrieved by using the To_String function.

with Ada.Text_IO;
with Util.Serialize.IO.JSON;
with Util.Streams.Texts;
procedure Serialize is
   Output : aliased Util.Streams.Texts.Print_Stream;
   Stream : Util.Serialize.IO.JSON.Output_Stream;
begin
   Output.Initialize (Size => 10000);
   Stream.Initialize (Output => Output'Unchecked_Access);
   Write (Stream);
   Ada.Text_IO.Put_Line (Util.Streams.Texts.To_String (Output));
end Serialize;

The Write procedure described above produces the following JSON content:

{"person":{"name":"Harry Potter","gender":"male","age": 17}}

Complete source example: serialize.adb

XML Serialization

Switching to an XML serialization is easy: replace JSON by XML in the package to use the XML serializer instead.

with Ada.Text_IO;
with Util.Serialize.IO.XML;
with Util.Streams.Texts;
procedure Serialize is
   Output : aliased Util.Streams.Texts.Print_Stream;
   Stream : Util.Serialize.IO.XML.Output_Stream;
begin
   Output.Initialize (Size => 10000);
   Stream.Initialize (Output => Output'Unchecked_Access);
   Write (Stream);
   Ada.Text_IO.Put_Line (Util.Streams.Texts.To_String (Output));
end Serialize;

This time, the same Write procedure produces the following XML content:

<person><name>Harry Potter</name><gender>male</gender><age>17</age></person>

Complete source example: serialize_xml.adb