object map list object classname map classname list reference @ int-index-previous-object-value map {} { members } members pair pair , members pair string : value symbol : value number : value list [] [ elements ] elements value value , elements string '' ' chars ' chars char char chars char any-printable-ASCII-character-except-'-"-or-\ \' \" \\ \/ \b \f \n \r \t \u four-hex-digits symbol # chars-limited # ' chars ' chars-limited char-limited char-limited chars-limited char-limited
STON: a Smalltalk Object Notation a-z A-Z 0-9 - _ . / classname uppercase-alpha-char alphanumeric-char number int int frac int exp int frac exp int digit digit1-9 digits - digit - digit1-9 digits frac . digits exp e digits digits digit digit digits e e e+ e- E E+ E- 134
C H A P T E R
10
Serializing Complex Objects with
Fuel
Fuel is a fast open-source general-purpose binary object serialization frame- work developed by Mariano Martinez-Peck, Martìn Dias and Max Leske. It is robust and used in many industrial cases. A fundamental reason for the cre- ation of Fuel was speed: while there is a plethora of frameworks to serialize objects based on recursive parsing of the object graphs in textual format as XML, JSON, or STON, these approaches are often slow. (For JSON and STON see also Chapters STON and NeoJSON.)
Part of the speed of Fuel comes from the idea that objects are loaded more often than stored. This makes it worth to spend more time while storing to yield faster loading. Also, its storage scheme is based on the pickle format that puts similar objects into groups for efficiency and performance. As a re- sult, Fuel has been shown to be one of the fastest object loaders, while still being a really fast object saver. Moreover, Fuel can serialize nearly any ob- ject in the image, it can even serialize a full execution stack and later reload it!
The main features of Fuel are as follows: • It has an object-oriented design. • It does not need special VM-support. • It is modularly packaged.
• It can serialize/materialize not only plain objects but also classes, traits, methods, closures, contexts, packages, etc.
Serializing Complex Objects with Fuel
• It is very customizable: you can ignore certain instance variables, sub- stitute objects by others, define pre and post serialization and materi- alization actions, etc.
• It supports class renaming and class reshaping.
• It has good test coverage and a large suite of benchmarks.
10.1
General Information
Fuel has been developed and maintained over the years by the following peo- ple: Martin Dias, Mariano Martinez Peck, Max Leske, Pavel Krivanek, Tristan Bourgois and Stéphane Ducasse (as PhD advisor and financer).
The idea of Fuel was developed by Mariano Martinez Peck based on the work by Eliot Miranda who worked on the ”parcels” implementation for Visual- Works. Eliot’s work again was based on the original ”parcels” implementa- tion by David Leib. ”Parcels” demonstrates very nicely that the binary pickle format can be a good alternative to textual storage and that grouping of ob- jects makes a lot of sense in object oriented systems.
Before going into details we present the ideas behind Fuel and it’s main fea- tures and give basic usage examples.
Goals
Concrete Fuel doesn’t aspire to have a dialect-interchange format. This makes it possible to serialize special objects like contexts, block closures, exceptions, compiled methods and classes. Although there are ports to other dialects, most notably Squeak, Fuel development is Pharo- centric.
Flexible Depending on the context, there can be multiple ways of serializing the same object. For example, a class can be considered either a global or a regular object. In the former case, references to the class will be encoded by name and the class is expected to be part of the environ- ment upon materialization; in the latter case, the class will be encoded in detail, with its method dictionary, etc.
Fast Fuel has been designed for performance. Fuel comes with a complete benchmark suite to help analyse the performance with diverse sample sets, as well as to compare it against other serializers. Fuel’s pickling algorithm achieves outstanding materialization performance, as well as very good serialization performance, even when compared to other binary formats such as ImageSegment.
Object-Oriented A requirement from the onset was to have a good object- oriented design and to avoid special support from the virtual machine.
10.1 General Information
Maintainable Fuel has a complete test suite (over 600 unit tests), with a high degree of code coverage. Fuel also has well-commented classes and methods.
Installation and Demo
Fuel 1.9 is available by default in Pharo since version 2.0 of Pharo. Therefore you do not need to install it. Thedefault packageswork out of the box in Pharo 1.1.1, 1.1.2, 1.2, 1.3, 1.4, 2.0, 3.0 and 4.0 and Squeak 4.1, 4.2, 4.3, 4.4, 4.5. The stable version at the time of writing is 1.9.4.
Open theTranscriptand execute the code below in aPlayground. This example serializes a set, the defaultTranscript(which is a global) and a block. On materialization it shows that
• the set is correctly recreated,
• the globalTranscriptis still the same instance (hasn’t been modified) • and the block can be evaluated properly.
| arrayToSerialize materializedArray | arrayToSerialize :=
Array
with: (Set with: 42) with: Transcript
with: [ :aString | Transcript show: aString; cr ]. "Store (serialize)"
FLSerializer serialize: arrayToSerialize toFileNamed: 'demo.fuel'. "Load (materialize)"
materializedArray := FLMaterializer materializeFromFileNamed: 'demo.fuel'.
Transcript
show: 'The sets are equal: ';
show: arrayToSerialize first = materializedArray first; cr;
show: 'But not the same: ';
show: arrayToSerialize first ~~ materializedArray first; cr;
show: 'The global value Transcript is the same: ';
show: arrayToSerialize second == materializedArray second; cr.
materializedArray third
value: 'The materialized block closure can be properly evaluated.'.
Serializing Complex Objects with Fuel
Some Links
• The home page ishttp://rmod.inria.fr/web/software/Fuel. • The source code is athttp://smalltalkhub.com/#!/~Pharo/Fuel. • The CI job is athttps://ci.inria.fr/pharo-contribution/job/Fuel-Stable/.
10.2
Getting Started
Basic Examples
Fuel offers some class-side messages to ease more common uses of serializa- tion (theserialize:toFileNamed:message ) and materialization (the mes- sagematerializeFromFileNamed:). The next example writes to and reads from a file:
FLSerializer serialize: 'stringToSerialize' toFileNamed: 'demo.fuel'. materializedString := FLMaterializer materializeFromFileNamed:
'demo.fuel'.
Fuel also provides messages for storing into aByteArray, namely the mes- sagesserializeToByteArray:andmaterializeFromByteArray:. This can be interesting, for example, for serializing an object graph as a blob of data into a database when using Voyage (see Chapter Voyage).
anArray := FLSerializer serializeToByteArray: 'stringToSerialize'. materializedString := FLMaterializer materializeFromByteArray:
anArray.
FileStream
In the following example we work with file streams. Note that the stream needs to be set to binary mode:
'demo.fuel' asFileReference writeStreamDo: [ :aStream | FLSerializer newDefault
serialize: 'stringToSerialize' on: aStream binary ].
'demo.fuel' asFileReference readStreamDo: [ :aStream | materializedString := (FLMaterializer newDefault
materializeFrom: aStream binary) root ].
In this example, we are no longer using the class-side messages. Now, for bothFLSerializerandFLMaterializer, we first create instances by send- ing thenewDefaultmessage and then perform the desired operations. As we will see in the next example, creating the instances allows for more flexibil- ity on serialization and materialization.
10.2 Getting Started
Compression
Fuel does not care to what kind of stream it writes its data. This makes it easy to use stream compressors. An example of use is as follows:
'number.fuel.zip' asFileReference writeStreamDo: [ :aFileStream | |gzip|
aFileStream binary.
gzip := GZipWriteStream on: aFileStream.
FLSerializer newDefault serialize: 123 on: gzip. gzip close ].
'number.fuel.zip' asFileReference readStreamDo: [ :aFileStream | |gzip|
aFileStream binary.
gzip := GZipReadStream on: aFileStream.
materializedString := (FLMaterializer newDefault materializeFrom: gzip) root.
gzip close ].
Showing a Progress Bar
Sometimes it is nice to see progress updates on screen. Use the message
showProgressin this case. The progress bar functionality is available from theFuelProgressUpdatepackage, so load that first:
Gofer it
url: 'http://smalltalkhub.com/mc/Pharo/Fuel/main'; package: 'ConfigurationOfFuel';
load.
(ConfigurationOfFuel project version: #stable) load: 'FuelProgressUpdate'.
The following example uses the messageshowProgressto display a progress bar during operations.
'numbers.fuel' asFileReference writeStreamDo: [ :aStream | FLSerializer newDefault
showProgress;
serialize: (1 to: 200000) asArray on: aStream binary ].
'numbers.fuel' asFileReference readStreamDo: [ :aStream | materializedString :=
(FLMaterializer newDefault showProgress;
Serializing Complex Objects with Fuel