Working with nested lists

It is often necessary to work with nested lists, that is - lists whose elements themselves are lists. We have seen simple examples of such lists already. Let me emphasize that in general such lists are not identical to multidimensional arrays but in fact much more general, because the lengths of sublists at each level can be different. The only thing we can say about the general nested list is that it represents some tree.

Here we will consider several special-purpose commands which were designed for efficient processing of some special types of such nested lists.

Ÿ 3.8.1 Partition

This command is used to "cut" or "slice" some list into (possibly overlapping) pieces. In its simplest form, it has a format Partition[list, size, shift]. It cuts the list into pieces with the length <size>, and shifted one with respect to another by <shift>. If the <shift> parameter is not given, the list is cut into non-overlapping pieces. For example:

Ÿ 3.8.1.1 A simple example

testlist=Table@Sqrt@iD, 8i, 1, 10<D

:1, 2 , 3 , 2, 5 , 6 , 7 , 2 2 , 3, 10>

Partition@testlist, 3D

::1, 2 , 3 >,:2, 5 , 6>,: 7 , 2 2 , 3>>

Partition@testlist, 7D

::1, 2 , 3 , 2, 5 , 6 , 7>>

In the last example, the remaining piece had a size smaller than 7, so it was "eaten up". Now we will partition with overlaps:

Partition@testlist, 7, 1D

::1, 2 , 3 , 2, 5 , 6 , 7>,: 2 , 3 , 2, 5 , 6 , 7 , 2 2 >, : 3 , 2, 5 , 6 , 7 , 2 2 , 3>,:2, 5 , 6 , 7 , 2 2 , 3, 10>>

Ÿ 3.8.1.2 An example of practical use: computation of the moving average in a list.

This example is based on a similar discussion in Wagner’ 96.

The problem

The m-moving average for a list is an average which is obtained by averaging every element in a list with

<m> neighbors to the right and to the left (which means that this quantity is only defined for points (elements) having at least m neighbours both to the left and to the right). Thus, moving average is actually a list of such averages, of the length <len>-2m, where <len> is a length of an initial list.

<m> neighbors to the right and to the left (which means that this quantity is only defined for points (elements) having at least m neighbors both to the left and to the right). Thus, moving average is actually a list of such averages, of the length <len>-2m, where <len> is a length of an initial list.

Developing a solution

To solve our problem, we will first define an auxiliary function which will count the average of a list of numbers. However, it will turn out that our function will also work on a list of lists of numbers, this time summing entire lists (with the same number of elements) together, which we will use. So:

Clear@listAverageD;

listAverage@x_ListD:= Apply@Plus, xD Length@xD;

The expression Apply[Plus,x] computes the sum of elements in the list and its meaning will be explained in chapter V.

Now we will define another auxiliary function:

Clear@neighborListsD;

neighborLists@x_List, m_IntegerD:=

Partition@x, Length@xD -2*m, 1D; For example:

neighborLists@testlist, 1D

::1, 2 , 3 , 2, 5 , 6 , 7 , 2 2>, : 2 , 3 , 2, 5 , 6 , 7 , 2 2 , 3>, : 3 , 2, 5 , 6 , 7 , 2 2 , 3, 10>>

Let us now realize that the middle list represents a list of "middle points", and the first and the last list represent here lists of closest "neighbors" for these middle points. Thus, the only thing left to do is to use listAverage on this result:

listAverage@neighborLists@testlist, 1DD :1

3 J1 + 2 + 3 N, 1

3 J2 + 2 + 3N, 1

3 J2 + 3 + 5N, 1

3 J2 + 5 + 6N, 1

3 J 5 + 6 + 7N, 1

3 J2 2 + 6 + 7 N, 1

3 J3 +2 2 + 7N, 1

3 J3 +2 2 + 10N>

Packaging code to a function

Thus, our final function <movingAverage>will look like:

Clear@movingAverage, neighborLists, listAverageD; neighborLists@x_List, m_IntegerD:=

Partition@x, Length@xD -2*m, 1D;

listAverage@x_ListD:= Apply@Plus, xD Length@xD; movingAverage@x_List, m_IntegerD:=

listAverage@neighborLists@x, mDD;

For example, here we find the moving average with two neighbors on each side:

movingAverage@testlist, 2D

:1

5 J3 + 2 + 3 + 5N, 1

5 J2 + 2 + 3 + 5 + 6N, 1

5 J2 + 3 + 5 + 6 + 7 N, 1

5 J2 +2 2 + 5 + 6 + 7N, 1

5 J3 +2 2 + 5 + 6 + 7N, 1

5 J3 +2 2 + 6 + 7 + 10 N>

Using functional programming to eliminate auxiliary functions

With the help of the functional programming syntax, we can write this as a single function and eliminate the need in auxiliary functions altogether:

Clear@movingAverageD;

movingAverage@x_List, m_IntegerD:=

HPlusžž ðL Length@ðD&žPartition@x, Length@xD -2*m, 1D;

Check:

movingAverage@testlist, 2D :1

5 J3 + 2 + 3 + 5N, 1

5 J2 + 2 + 3 + 5 + 6N, 1

5 J2 + 3 + 5 + 6 + 7 N, 1

5 J2 +2 2 + 5 + 6 + 7N, 1

5 J3 +2 2 + 5 + 6 + 7N, 1

5 J3 +2 2 + 6 + 7 + 10 N>

A procedural version

Here is the procedural implementation of the same thing:

movingAverageProc@x_List, m_IntegerD:=

Module@8i, j, ln=Length@xD, aver, sum<, aver =Table@0, 8ln -2*m<D;

For@i=m +1, i<=ln -m, i ++, sum =0;

For@j=i -m, j£ i +m, j ++, sum=sum +x@@jDDD;

aver@@i -mDD =sum H2*m +1LD; averD;

Check:

movingAverageProc@testlist, 2D :1

5 J3 + 2 + 3 + 5N, 1

5 J2 + 2 + 3 + 5 + 6N, 1

5 J2 + 3 + 5 + 6 + 7 N, 1

5 J2 +2 2 + 5 + 6 + 7N, 1

5 J3 +2 2 + 5 + 6 + 7N, 1

5 J3 +2 2 + 6 + 7 + 10 N>

Efficiency comparison

The problem with the procedural version is not just that the code is longer, but also that it is more error prone (array bounds, initialization of variables etc). On top of that, it turns out to be far less efficient. Let us compare the efficiency on large lists:

Timing@movingAverage@Range@10 000D, 10D;D 80.016 Second, Null<

Timing@movingAverageProc@Range@10 000D, 10D;D 81.172 Second, Null<

Here we have a 100 times difference (for this length of the list)! And moreover, this is not a constant factor, but the difference will increase further with the length of the list. Of course, in procedural lan-guages such as C the latter implementation is natural and fast. Not so in Mathematica. However, one can still obtain the code which will be concise, fast and elegant at the same time, with the use of functional programming methods.

Clear@testlistD;

Ÿ 3.8.2 Transpose

This is one of the most useful commands. It has this name since for matrices, which are represented as 2-dimensional lists of lists, it performs the transposition operation. However, we are not forced to always interpret the two-dimensional array as a matrix, especially if it is combined from elements of different types. Then it turns out that the number of useful things one can do with Transpose is much larger. But let us start with the numeric lists: say we have a given list of lists of some elements (they may be lists them-selves, but this does not matter for us):

Ÿ 3.8.2.1 Simple example: transposing a simple matrix testlist=Table@i +j, 8i, 1, 2<, 8j, 1, 3<D 882, 3, 4<, 83, 4, 5<<

Then,

Transpose@testlistD 882, 3<, 83, 4<, 84, 5<<

Ÿ 3.8.2.2 Example: transposing a matrix of lists Another example:

testlist=Table@8i, j<, 8i, 1, 2<, 8j, 1, 3<D

8881, 1<, 81, 2<, 81, 3<<, 882, 1<, 82, 2<, 82, 3<<<

This is a 2-dimensional array of lists.

Transpose@testlistD

8881, 1<, 82, 1<<, 881, 2<, 82, 2<<, 881, 3<, 82, 3<<<

Ÿ 3.8.2.3 Example: combining names with grades

Another example: we have results of some exam - the scores - as a first list, and last names of the students as another one. We want to make a single list of entries like {{student1,score1},...}.

Clear@names, scoresD;

names = 8"Smith", "Johnson", "Peterson"<; scores = 870, 90, 50<;

Then we do this:

Transpose@8names, scores<D

88Smith, 70<, 8Johnson, 90<, 8Peterson, 50<<

But we will get most out of Transpose when we get to functional programming, since Transpose is very frequently used there for efficient structure rearrangements. We will see many examples of its use in the chapters that follow.

Ÿ 3.8.3 Flatten

Ÿ

3.8.3 Flatten

This command is used to destroy the structure of nested lists, since it removes all internal curly braces and transforms any complicated nested list into a flat one. For example:

Ÿ 3.8.3.1 Simple example: flattening a nested list

testlist=Table@8i, j<, 8i, 1, 2<, 8j, 1, 3<D

8881, 1<, 81, 2<, 81, 3<<, 882, 1<, 82, 2<, 82, 3<<<

Flatten@testlistD

81, 1, 1, 2, 1, 3, 2, 1, 2, 2, 2, 3<

Ÿ 3.8.3.2 Flattening down to a given level

One can make Flatten more "merciful" and selective by instructing it to destroy only braces up to (or, more precisely, down to) a certain level in an expression. The level up to which the "destruction" has to be performed is given to Flatten as an optional second parameter. For instance:

Flatten@testlist, 1D

881, 1<, 81, 2<, 81, 3<, 82, 1<, 82, 2<, 82, 3<<

Example: flattening a nested list level by level

Another example:

testlist=Table@8i, j, k<, 8i, 1, 2<, 8j, 1, 2<, 8k, 1, 3<D

88881, 1, 1<, 81, 1, 2<, 81, 1, 3<<, 881, 2, 1<, 81, 2, 2<, 81, 2, 3<<<, 8882, 1, 1<, 82, 1, 2<, 82, 1, 3<<, 882, 2, 1<, 82, 2, 2<, 82, 2, 3<<<<

Flatten@testlist, 1D

8881, 1, 1<, 81, 1, 2<, 81, 1, 3<<, 881, 2, 1<, 81, 2, 2<, 81, 2, 3<<, 882, 1, 1<, 82, 1, 2<, 82, 1, 3<<, 882, 2, 1<, 82, 2, 2<, 82, 2, 3<<<

Flatten@testlist, 2D

881, 1, 1<, 81, 1, 2<, 81, 1, 3<, 81, 2, 1<, 81, 2, 2<, 81, 2, 3<, 82, 1, 1<, 82, 1, 2<, 82, 1, 3<, 82, 2, 1<, 82, 2, 2<, 82, 2, 3<<

Flatten@testlist, 3D

81, 1, 1, 1, 1, 2, 1, 1, 3, 1, 2, 1, 1, 2, 2, 1, 2,

3, 2, 1, 1, 2, 1, 2, 2, 1, 3, 2, 2, 1, 2, 2, 2, 2, 2, 3<

In practice, most frequently one uses either Flatten[expr] to get a completely flat list, or Flatten[expr,1]

to remove some internal curly braces which were needed at some intermediate steps but not anymore.

Ÿ 3.8.3.3 Application: a computation of quadratic norm of a tensor of arbitrary rank (vector, matrix etc).

Ÿ

3.8.3.3 Application: a computation of quadratic norm of a tensor of arbitrary rank (vector, matrix etc).

The problem and the solution

Here we will show how the use of Flatten can dramatically simplify the computation of the norm of a tensor of arbitrary rank. What may be surprising is that we will not need the rank of the tensor as a sepa -rate parameter. So, we wil start with the code:

Clear@tensorNormD;

tensorNorm@x_ListD:=Sqrt@PlusžžFlatten@x ^ 2DD;

It turns out that this tiny code is all what is needed to solve the problem in all generality.

Code dissection

Let us use an example to show how it works. This will be our test matrix:

testmatr=Table@i +j, 8i, 1, 3<, 8j, 1, 3<D 882, 3, 4<, 83, 4, 5<, 84, 5, 6<<

The norm is the square root of sum of the squares of all matrix elements. First, we will use the fact that arithmetic operations such as raising to some power, can be used on entire lists, because they are automati-cally threaded over the elements of the list (functions which have this property are said to be Listable).

Thus, we first square all the elements:

testmatr ^ 2

884, 9, 16<, 89, 16, 25<, 816, 25, 36<<

Since we don’t care which elements are where but just need to sum them all, we will use Flatten to remove the internal curly braces:

Flatten@testmatr ^ 2D

84, 9, 16, 9, 16, 25, 16, 25, 36<

Now we have to some all the elements, and as we saw already this can be done for instance with Plus@@

construction:

PlusžžFlatten@testmatr ^ 2D 156

Finally, we have to take a square root:

Sqrt@PlusžžFlatten@testmatr ^ 2DD 2 39

And we arrive at the code of our function. We see that the function works well on a tensor of any rank without modifications! It would be hard to do this without Flatten, and in particular, in languages like C we would need nested loops to accomplish this (in C, there is also a technique called flattening an array, which consists in exploiting the row-major order in which it is stored in memory and going through the multidimensional array with just a pointer to an integer (or whatever is the type of the smallest array element). Although it usually works, it will be illegal if one wants to strictly adhere to the C standard).

we would need nested loops to accomplish this (in C, there is also a technique called flattening an array, which consists in exploiting the row-major order in which it is stored in memory and going through the multidimensional array with just a pointer to an integer (or whatever is the type of the smallest array element). Although it usually works, it will be illegal if one wants to strictly adhere to the C standard).

Clear@tensorNorm, testmatrD;

Ÿ 3.8.3.4 Application - (relatively) fast list generation with Flatten

As we already mentioned, generating lists straightforwardly in loops is perhaps the worst way to do it, in terms of efficiency. One can use Flatten to speed-up this process considerable. Say, we want to generate a list from 1 to 10 (which is easiest to do, of course, by just using Range[10]). We can do it in the following fashion:

Step 1. We generate a nested list (this type of lists are also called linked lists in Mathematica):

For@testlist=8<; i=1, i£ 10, i ++, testlist=8testlist, i<D; testlist

88888888888<, 1<, 2<, 3<, 4<, 5<, 6<, 7<, 8<, 9<, 10<

Step 2. We use Flatten:

Flatten@testlistD

81, 2, 3, 4, 5, 6, 7, 8, 9, 10<

Let us compare the execution time with the realization with Append described previously:

For@testlist=8<; i=1, i<5000, i ++, AppendTo@testlist, iDD;

Timing

80.25 Second, Null<

Now, with our new method:

HFor@testlist=8<; i=1, i<5000, i ++, testlist=8testlist, i<D; Flatten@testlistD;L Timing

80.016 Second, Null<

We see that the difference is about an order of magnitude at least. While even this method by itself is not the most efficient, we will later see how linked lists can be used in certain problems to dramatically improve efficiency.

Clear@testlistD;

In document Mathematica Programming - Advanced Intro (Page 77-84)