operator overloading algorithmic differentiation the classdifferentialnumber operator overloading

(1)

operator overloading

1 _{Computing with Differential Numbers}

algorithmic differentiation

the class DifferentialNumber operator overloading

2 _{Computing with Double Doubles}

the class DoubleDouble defining +, -, *, and / expression evaluation

3 _{Wrapping C Code with SWIG}

swig = simplified wrapper and interface generator

MCS 507 Lecture 8 Mathematical, Statistical and Scientific Software Jan Verschelde, 13 September 2019

(2)

operator overloading

(3)

computing with differential numbers

A differential number is a tuple df = (f , f′_{) of two doubles}

where f′ _{is the evaluated derivative of f .}

The operator overloading encodes the elementary derivative rules for addition, subtraction, multiplication, and division of two real numbers. As we evaluate any expression written in these four elementary operations, we also compute its derivative.

With a Python script we illustrate the forward mode of algorithmic differentiation.

Reference: section (c) on computing with differential numbers of Introduction to Numerical Analysis, pages 7 to 10, by Arnold Neuimaier, Cambridge 2001.

(4)

encoding differentiation rules

Let df = (f , f′_{) and dg = (g, g}′_),

where f and g are dependent on one single variable x. The four basic elementary operations are_{+, −, ⋆ and /,} defined as follows:

df + dg = (f , f′_{) + (g, g}′_{) = (f + g, f}′_{+ g}′₎

df − dg = (f , f′_{) − (g, g}′_{) = (f − g, f}′_{− g}′₎

df ⋆ dg = (f , f′_{) ⋆ (g, g}′_{) = (f ⋆ g, f}′_{⋆ g + f ⋆ g}′₎

df/dg = (f , f′_{)/(g, g}′_{) = (f /g, (f}′_{− (f /g) ∗ g}′_)/g)

Constants and the independent variable as differential numbers: Any constant c (independent of x) is represented as(c, 0). The independent variable x is represented as(x, 1).

(5)

evaluating derivatives without explicit formulas

Consider f(x) = (x − 1)(x + 3)

x + 2 at x0= 3. Recall the rules: (f , f′_{) ± (g, g}′_{) = (f ± g, f}′_{± g}′_),

(f , f′_{) ⋆ (g, g}′_{) = (f ⋆ g, f}′_{⋆ g + f ⋆ g}′_),

(f , f′_{)/(g, g}′_{) = (f /g, (f}′_{− (f /g) ∗ g}′_)/g).

We substitute the differential number(3, 1) in f : (f (3), f′_{(3)) =}

f((3, 1)) = ((3, 1) − 1) ⋆ ((3, 1) + 3) (3, 1) + 2 = (2, 1) ∗ (6, 1) (5, 1) = (12, 8) (5, 1) = (2.4, (8 − 2.4 ⋆ 1)/5) = (2.4, 1.12)

(6)

operator overloading

the class DifferentialNumber

operator overloading

(7)

the data attribute of the class

class DifferentialNumber(object): """

A differential number is a tuple df = (f, f’) of two doubles

where f’ is the evaluated derivative of f. """

def __init__(self, xv, xp=1): """

Initializes a differential number to the tuple (xv, xp).

"""

self.val = float(xv) self.der = float(xp)

By default, DifferentialNumber(x) for any x is(x, 1). Note the explicit conversion to type float.

(8)

the string respresentation

A differential number(f , f′_{) is represented as a tuple.}

def __str__(self): """

Returns the tuple representation of the differential number. """

(9)

operator overloading

the class DifferentialNumber

operator overloading

(10)

adding differential numbers

The rule df + dg = (f , f′_{) + (g, g}′_{) = (f + g, f}′_{+ g}′_{) is implemented}

by overriding the builtin operator+ defined by __add__. The self refers to the first operand,

while the second operand is given by the other argument. def __add__(self, other):

"""

Defines the addition of two differential numbers. """

return DifferentialNumber(self.val + other.val, \ self.der + other.der)

(11)

dealing with cases like x + 2

_.0

def __add__(self, other): """

Defines the addition of two differential numbers. """

if isinstance(other, float): return DifferentialNumber \

(self.val + other, self.der) else:

return DifferentialNumber \

(self.val + other.val, \ self.der + other.der)

(12)

reflected addition

The reflected (or swapped) addition happens when the first operand in +is not a DifferentialNumber but an ordinary number.

def __radd__(self, other): """

Addition when operand is not a DifferentialNumber as in 2.0 + x. """

result = self + other return result

When x is a DifferentialNumber, then x+y is executed as x.__add__(y), where x is self and y is other.

For x + y when x is not a DifferentialNumber, but y is a DifferentialNumber,

(13)

subtraction of two differential numbers

The operator_{− is defined by the __sub__() method:} def __sub__(self, other):

"""

Defines the subtraction of two differential numbers. """

if isinstance(other, float): return DifferentialNumber \

(self.val - other, self.der) else:

(self.val - other.val, \ self.der - other.der)

This implements the rule df _{− dg = (f , f}′_{) − (g, g}′_{) = (f − g, f}′_{− g}′₎

(14)

multiplying two differential numbers

The rule(f , f′_{) ⋆ (g, g}′_{) = (f ⋆ g, f}′_{⋆ g + f ⋆ g}′₎

is implemented by overriding the method __mul__(): def __mul__(self, other):

"""

Defines the product of two differential numbers. """

(self.val*other.val, \ self.der*other.val + \ self.val*other.der)

(15)

multiplication with a constant

For a constant c: (f , f′_{) ⋆ (c, 0) = (f ⋆ c, f}′_{⋆ c).}

def __mul__(self, other): """

Defines the product of two differential numbers. """ if isinstance(other, float): return DifferentialNumber \ (self.val*other, self.der*other) else: return DifferentialNumber (self.val*other.val, \ self.der*other.val + \ self.val*other.der)

(16)

dividing two differential numbers

We override the __truediv__() method to define division. Coding the rule(f , f′_{)/(g, g}′_{) = (f /g, (f}′_{− (f /g) ∗ g}′_)/g):

def __truediv__(self, other): """

Defines the division of two differential numbers. """

val = self.val/other.val return DifferentialNumber \

(val, (self.der \

(17)

dividing by a constant

For a constant c: (f , f′_{)/(c, 0) = (f /c, f}′_/c).

def __truediv__(self, other): """

Defines the division of two differential numbers. """ if isinstance(other, float): return DifferentialNumber \ (self.val/other, self.der/other) else: val = self.val/other.val return DifferentialNumber \ (val, (self.der \ - val*other.der)/other.val)

(18)

the main test program

def main(): """

Test on the expression ((x-1)*(x+3))/(x+2). """ dfx = DifferentialNumber(3, 1) fun = lambda x: ((x-1.0)*(x+3.0))/(x+2.0) print(’((x-1)*(x+3))/(x+2) at’, dfx, ’:’) dfy = fun(dfx) print(dfy) if __name__ == "__main__": main()

(19)

running the test

$ python differential_numbers.py ((x-1)*(x+3))/(x+2) at (3.0, 1.0) : (2.4, 1.1199999999999999)

(20)

extensions of differential numbers

Two exercises:

1 _{Extend the class DifferentialNumber so it works for}

expressions in two variables and a differential number df is

(f , fx, fy), where fx is the derivative with respect to the first variable

and fy is the derivative with respect to the second variable.

2 Extend the class DifferentialNumber to compute also the

(21)

numerical floats

Exercise:

3 We estimate the roundoff error of an arithmetical operation ,

for ∈ {+, −, ⋆, /} and floating-point numbers x and y

as x ey = (xy)(1 + ǫ), where e is the floating-point version of and 0_{≤ ǫ < ǫ}mach, for the machine precisionǫmach. The roundoff

error is bounded by|xy|ǫmach. For accumulated roundoff:

x0xe 1e· · · exn

| {z }

n operations

≈ (x0x1· · · xn)(1 + ǫ1+ ǫ2+ · · · + ǫn+ · · · ),

Truncating higher-order terms, the accumulated roundoff on n operations is estimated by_|x0x1· · · xn|nǫmach.

Define a class NumericalFloat, overriding_{+, −, ⋆, and /, to} also compute the upper bound on the roundoff for each operation. The accumulated roundoff is stored as an extra data attribute with each float.

(22)

operator overloading

the class DoubleDouble

defining +, -, *, and / expression evaluation

(23)

wrapping double double arithmetic

The QD library provides double double and quad double arithmetic. To bring the functionality of the QD library into Python,

we define an extension module in three steps:

1 _{Write wrapper functions that return Python objects.} 2 _{Define the table of methods that will be exported.} 3 Setup the compilation and linking instructions.

The result is a shared object file which can be imported into a Python session.

(24)

the string representation in C

Consider the prototype: #include <stdio.h> #include <stdlib.h> #include <qd/c_dd.h> #include <Python.h>

char *str ( double hi, double lo ); /*

* DESCRIPTION :

* Returns the string representation of * the double double with high part in hi * and low part in lo. */

(25)

the wrapped string representation

static PyObject *doubleDouble_str ( PyObject *self, PyObject *args ) {

PyObject *result; double hi, lo; char *r; if(!PyArg_ParseTuple(args,"dd",&hi,&lo)) return NULL; r = str(hi,lo); result = (PyObject*)Py_BuildValue("s",r); return result; }

(26)

the table of exported methods

static PyMethodDef double_double_methods[] = {

{ "str" , doubleDouble_str , METH_VARARGS,

"returns the string representation of a double double" } , ...

{ NULL , NULL, 0, NULL } , };

static struct PyModuleDef doubleDoubleModule = { PyModuleDef_HEAD_INIT,

"double_double",

NULL, /* no module documentation */ -1, double_double_methods }; PyMODINIT_FUNC PyInit_doubleDouble(void) { return PyModule_Create(&doubleDoubleModule); }

(27)

compilation instructions

The QD_ROOT and QD_LIB define the locations of the code of the QD library and its installed library. The PYTHON36 and PYTHON36lib define the locations of the Python.h and the libraries.

The makefile on a Mac OS X computer: double_double.o: double_double.c g++ -c -I$(QD_ROOT)/include -I$(PYTHON36) \ -I$(PYTHON36)/Include double_double.c doubleDouble.so: double_double.o g++ -dynamiclib -o doubleDouble.so \ double_double.o $(QD_LIB)/libqd.a \ /usr/lib/libc++.dylib /usr/lib/libstdc++.dylib \ $(PYTHON36lib)/libpython3.6m.dylib -lm

(28)

operator overloading

To add two double doubles x and y,

instead of z = doubleDouble.add(x[0],x[1],y[0],y[1]) we would like to write z = x + y.

We will define the class DoubleDouble() and export the method __add__.

Defining the method __add__ allows to use the +

operator on any two elements of the class DoubleDouble. The + is a binary operator:

1 the self is the first operand of +, 2 _other_{is the second operand of +.}

The definition starts with def __add__(self,other):.

Subtraction, multiplication, and division are defined respectively by __sub__, __mul__, and __div__.

(29)

data attributes of the class DoubleDouble

Importing the module doubleDouble that wraps some of the basic functionality of the QD library:

import doubleDouble

class DoubleDouble(): """

Wraps some of the functionality of the QD library to perform double double arithmetic.

"""

def __init__(self, hi=0.0, lo=0.0): """

A double double consists of a high and a low part. """

self.high = hi self.low = lo

(30)

string representations and deep copy

def __str__(self): """

Returns the string representation of a double double. """

return doubleDouble.str(self.high, self.low)

def __repr__(self): """

Returns the representation of a double double. """

return self.__str__()

def copy(self): """

Returns a copy of the double double. """

(31)

operator overloading

the class DoubleDouble

defining +, -, *, and /

expression evaluation

(32)

addition and subtraction

def __add__(self, other): """

Returns the sum of two double doubles. """

z = doubleDouble.add(self.high, self.low, \ other.high, other.low) return DoubleDouble(z[0], z[1])

def __sub__(self, other): """

Returns the difference of self and the other. """

z = doubleDouble.sub(self.high, self.low, \ other.high, other.low) return DoubleDouble(z[0], z[1])

(33)

multiplication and division

def __mul__(self, other): """

Returns the product of two double doubles. """

z = doubleDouble.mul(self.high, self.low, \ other.high, other.low) return DoubleDouble(z[0], z[1])

def __div__(self, other): """

Returns the division of self by the other. """

z = doubleDouble.div(self.high, self.low, \ other.high, other.low) return DoubleDouble(z[0], z[1])

(34)

computing powers

To compute x**5, we overload the __pow__ method: def __pow__(self,n):

"""

Returns self to the power n. """

z = self.copy() for i in range(1,n):

z = z*self return z

(35)

a basic test on the arithmetic

def test(): """

Basic test on arithmetical operations. """

hi = 1.4142135623730951 lo = -9.6672933134529147e-17

x = DoubleDouble(hi, lo) # defines the sqrt(2)

print(’x = sqrt(2) =’, str(x)) y = x**2 print(’x*x =’, y) z = y/x print(’x*x/x =’, z) u = x+x print(’x+x =’, u) two = DoubleDouble(2.0) print(’2*x =’, two*x) v = u-x print(’x+x-x =’, v)

(36)

running the test

Adding to the end of double_double.py the line if __name__=="__main__": test() so we can run $ python double_double.py x = sqrt(2) = 1.4142135623730950488016887242096 x*x = 2.0000000000000000000000000000000 x*x/x = 1.4142135623730950488016887242096 x+x = 2.8284271247461900976033774484193 2*x = 2.8284271247461900976033774484193 x+x-x = 1.4142135623730950488016887242096

(37)

Newton’s method for

√

2

def newton4sqrt2(): """

Applies Newton’s method to approximate sqrt(2). """ x = DoubleDouble(2.0) print(’step 0 :’, x) for i in range(1,9): z = x**2 z = z + DoubleDouble(2.0) z = z/x z = DoubleDouble(0.5)*z x = z.copy() print(’step’, i, ’:’, x)

(38)

operator overloading

the class DoubleDouble defining +, -, *, and /

expression evaluation

(39)

a motivating example for interval arithmetic

Problem: Evaluate f(x, y) =

(333.75 − x2)y6+ x2(11x2y2_{− 121y}4_{− 2) + 5.5y}8+ x/(2y) at(77617, 33096).

An example of Stefano Taschini: Interval Arithmetic: Python

Implementation and Applications. In the Proceedings of the 7th Python

in Science Conference (SciPy 2008).

(40)

evaluating an expression

print(’checking the motivating interval arithmetic example’) f = lambda x,y: (DoubleDouble(333.75) - x**2)*y**6 \

+ x**2*(DoubleDouble(11)*x**2*y**2 \

- DoubleDouble(121)*y**4 - DoubleDouble(2)) \ + DoubleDouble(5.5)*y**8 + x/(DoubleDouble(2)*y); a = 77617; b = 33096

z = f(DoubleDouble(a),DoubleDouble(b))

# to check the answer with SymPy : import sympy as sp x,y = sp.var(’x,y’) g = (sp.Rational(33375)/100 - x**2)*y**6 \ + x**2*(11*x**2*y**2 - 121*y**4 - 2) \ + sp.Rational(55)/10*y**8 \ + sp.Rational(1)*x/(2*y);

print(’evaluating’, g, ’at’, (a,b)) e = sp.Subs(g,(x,y),(a,b)).doit()

(41)

script continued and running the code

e15 = e.evalf(15)

print(’numerical value :’, z)

print(’exact value :’, e, ’~’, e15) print(’error :’, abs(e15 - z.high))

We run the script as follows:

$ python double_double_eval.py

checking the motivating interval arithmetic example

evaluating x**2*(11*x**2*y**2 - 121*y**4 - 2) + x/(2*y) + 11 numerical value : 1.17260394005317863185883490452018e+00 exact value : -54767/66192 ~ -0.827396059946821

error : 2.00000000000000 $

Exercise:

(42)

operator overloading

(43)

Simplified Wrapper and Interface Generator

Available at www.swig.org:

SWIG is a software development tool that connects programs written in C and C++ with high-level programming languages. SWIG is used with different types of target languages including scripting languages such as Perl, PHP, Python, Tcl and Ruby. SWIG is most commonly used to create high-level interpreted or compiled programming environments, user interfaces, and as a tool for testing and prototyping C/C++ software.

SWIG is typically used to parse C/C++ interfaces and generate the ’glue code’ required for the above target languages to call into the C/C++ code.

SWIG is free software and the code that SWIG generates is compatible with both commercial and non-commercial projects. We will work out the example of the tutorial.

(44)

example.c

from www.swig.org/tutorial.html

#include <time.h> double My_variable = 3.0; int fact(int n) { if (n <= 1) return 1; else return n*fact(n-1); }

int my_mod(int x, int y) { return (x%y); } char *get_time() { time_t ltime; time(&ltime); return ctime(&ltime); }

(45)

example.i

from www.swig.org/tutorial

%module example %{

/* put header files here or function declarations */ extern double My_variable;

extern int fact(int n);

extern int my_mod(int x, int y); extern char *get_time();

%}

extern double My_variable; extern int fact(int n);

extern int my_mod(int x, int y); extern char *get_time();

(46)

the makefile

The makefile defines the compilation instructions, on RHEL: PYTHON=/usr/include/python2.6

example:

swig -python example.i

gcc -c -fPIC example.c example_wrap.c -I$(PYTHON) gcc -shared example.o example_wrap.o \

-o _example.so

Executing: $ make example

swig -python example.i

gcc -c -fPIC example.c example_wrap.c -I/usr/include/python2.6 gcc -shared example.o example_wrap.o -o _example.so

(47)

input and output files

Two input files:

1 _example.c_{with C code,} 2 _example.i_{is the interface file.}

To compile successfully, we must know

1 _{the location of the Python header files,}

2 _{proper linking options, e.g.: -framework Python on Mac.}

Four output files:

1 _{example_wrap.c}_{generated C code,} 2 _{example_wrap.o}compiled object file, 3 _{_example.so}_{shared object file, and} 4 _example.py_{Python code for wrapper.}

(48)

Summary and Additional Exercises

Operator overloading makes double double arithmetic natural. Another benefit of operator overloading is to add properties of the computed results, e.g.: derivatives, roundoff errors, operation counts.

Additional exercises:

5 _{Wrap the basic operations for quad double arithmetic of the QD}

library and make the class QuadDouble. Use the class to evaluate the expression of the motivating example for interval arithmetic.

6 _{Using inheritance extend the class DoubleDouble with an extra}

field count to count the number of arithmetical operations performed to compute a double double. For example, if two constants have counts m and n, their sum has count m+ n.