Session 5x: Bonus material
Johan Koskinen http://www.ccsr.ac.uk/staff/jk.htm!
The Social Statistics Discipline Area, School of Social Sciences
Mitchell Centre for Network Analysis
Workshop: Mon-Fri, 7-11 July 2014 Advanced Meths SNA, Manchester
Session 5x: Bonus material
q Bayesian analysis q Missing data
q Snowball sampled data
q Fitting ERGM to LARGE data sets q Spatial Embedded networks
q Multilevel ERGM
q Longitudinal ERGM
Bayesian analysis in MPNet
Bayesian inference (in MPNet):
Fishermen
Bayesian estimation
Go back into Select
parameters, start afresh by clearing all and then select
edge, ASA, ATA
BE PATIENT. Bayesian estimation can be slower (we are working on automation).
Quick MCMC settings for Bayesian
¡ We need a slightly large multiplication factor than for non-‐Bayesian estimation
¡ Maximum lag should be chosen to be roughly the lag where SACF is 0 (in order for ESS to be correct) – roughly 200-‐400
¡ If model is good we can use Pre-‐tuning only to get good initial values
§ The objective is to get high acceptance rate around .85
§ Run number of small MCMC sample sizes and press update
¡ When pre-‐tuning not too bad check ‘Nonconditional simulation’ and press update (the latter to start in a better place and get proposal covariance)
§ The objective is to get acceptance rate around .25 and SACF around lag 200 small and ESS large
§ If acceptance rate too small (say smaller than .15) reduce Proposal scaling (e.g.
divide by 2); if too large (say greater than .45) increase Proposal scaling (e.g.
multiply by 2)
§ Once SACF at large lags (say 100 or 200) is low (say, around .1) you can Improve the ESS by making the MCMC sample size bigger
¡ If you have a good run and want the perfect run read in ‘Covariance file’
Bayesian analysis in MPNet
¡ Set
§ multiplication factor to 80
§ Scale 0.005
§ MCMC sample size 1000
§ Max lag 200
§ Scaled identity
¡ After run
§ Note that Inverse D matrix is diagonal
EdgeA
Time
ts(output[, k + 1])
0 200 600 1000
-3.6-3.4-3.2-3.0-2.8
ASA
Time
ts(output[, k + 1])
0 200 600 1000
-0.3-0.2-0.10.0
ATA
Time
ts(output[, k + 1])
0 200 600 1000
0.00.20.40.60.8
Inverse D matrix:
0.0010 0.0000 0.0000
0.0000 0.0010 0.0000
0.0000 0.0000 0.0010
Acceptance rate: 0.42 Estimation results
Effects Lambda PostMean Stddev
EdgeA 2.0000 -3.2257 0.237 *
ASA 2.0000 -0.1862 0.094
ATA 2.0000 0.7372 0.139 *
SACF
Effect 10 30 190 ESS(200)
EdgeA 0.973 0.914 0.265 4
ASA 0.905 0.781 0.412 4
ATA 0.802 0.456 -0.053 12
This run just got us close to where we want to be
Bayesian analysis in MPNet
¡ Set
§ multiplication factor to 80
§ Scale 0.005
§ MCMC sample size 1000
§ Max lag 200
§ Scaled identity
¡ After run
§ Note that Inverse D matrix is diagonal
§ Press Update
§ MCMC sample size 5500; Parameter burnin 500
§ Proposal scaling 0.01
§ Nonconditional simulation
This run just got us close to where we want to be We want to draw values roughly here BUT more efficiently (by setting a better Proposal variance than the diagonal)
Bayesian analysis in MPNet
¡ Re-‐run with longer ‘moves’
§ Set Proposal scaling 1.50
§ rerun
-‐> It is moving around really well BUT it takes too small steps
(acceptance ratio: 0.93)
Inverse D matrix:
0.0015 -0.0005 0.0001
-0.0005 0.0002 -0.0001
0.0001 -0.0001 0.0001
Acceptance rate: 0.93 Estimation results
Effects Lambda PostMean Stddev
EdgeA 2.0000 -3.6399 0.385 *
ASA 2.0000 -0.0596 0.138
ATA 2.0000 0.7484 0.081 *
SACF
Effect 10 30 190 ESS(200)
EdgeA 0.952 0.871 0.470 20
ASA 0.959 0.887 0.499 20
ATA 0.955 0.870 0.370 22
EdgeA
Time
ts(output[, k + 1])
0 2000 4000
-4.5-4.0-3.5-3.0-2.5
ASA
Time
ts(output[, k + 1])
0 2000 4000
-0.4-0.20.00.2
ATA
Time
ts(output[, k + 1])
0 2000 4000
0.60.70.80.91.0
0 200 600 1000
0.00.20.40.60.81.0
ESS: 14
ACF
SACF for EdgeA
0 200 600 1000
-0.20.00.20.40.60.81.0
ESS: 12
ACF
SACF for ASA
0 200 600 1000
-0.20.20.61.0
ESS: 14
ACF
SACF for ATA
Bayesian analysis in MPNet
-‐> It is moving around really well AND it takes Nice LONG strides (acceptance ratio:
Acceptance rate: 0.34 Estimation results
Effects Lambda PostMean Stddev
EdgeA 2.0000 -3.5364 0.534 *
ASA 2.0000 -0.1223 0.185
ATA 2.0000 0.8167 0.122 *
SACF
Effect 10 … 190
ESS(200)
EdgeA 0.442 … -0.071 351
ASA 0.467 … -0.043 328
ATA 0.511 … 0.044 205
EdgeA
Time
ts(output[, k + 1])
0 1000 3000 5000
-5-4-3-2
ASA
Time
ts(output[, k + 1])
0 1000 3000 5000
-0.6-0.20.00.20.40.6
ATA
Time
ts(output[, k + 1])
0 1000 3000 5000
0.40.60.81.01.2
0 200 600 1000
0.00.20.40.60.81.0
ESS: 229
ACF
SACF for EdgeA
0 200 600 1000
0.00.20.40.60.81.0
ESS: 221
ACF
SACF for ASA
0 200 600 1000
0.00.20.40.60.81.0
ESS: 183
ACF
SACF for ATA
Effective sample size. We want these to be larger than 500, else the Stddev:s are misleading.
Here, as acceptance rate GOOD, rerun the estimation with larger MCMC sample size Here SACF is almost zero allready at lag 30!!!
Bayesian analysis in MPNet
The output file, [session name]_posterior_bayesian.txt contains the Bayesian posteriors.
EdgeA
Frequency
-5 -4 -3 -2
0400800
ASA
Frequency
-0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6
0400800
ATA
Frequency
0.4 0.6 0.8 1.0 1.2
0400800
-5 -4 -3 -2
-0.6-0.20.00.20.40.6
EdgeA
ASA
-5 -4 -3 -2
0.40.60.81.01.2
EdgeA
ATA
Missing data in MPNet
Session 4: More complex models
q The datafile miss20.txt is an 85X85 randomly simulated matrix with a density of 0.20. This will be a matrix equivalent 20%
missing data at random.
q The datafile fish_miss20.txt is the fishermen data except that all the 1’s in miss20.txt are set to zero.
In other words, fish_miss20.txt can be regarded as the fishermen’s network with 20% missing data (both 1’s and 0’s)
q Note that to use the missing data estimation in MPNET, you need to have an indicator matrix with 1 entered into every
missing cell, and all missing cells in the original data have to be entered as o’s.
Session 4: More complex models
In MPNET, under the Bayesian estimation tab:
• Enter the fish_miss20 file to be estimated
• Enter the miss20 as the missing indicators file,
• Select parameters and clear any previous parameter values (i.e.
start from 0)
• Conduct Bayesian estimation for an edge, AS and AT model. Use 3000 as the MCMC sample size.
Our results
Effects Lambda PostMean Stddev
EdgeA 2.0000 -4.4253 0.552 *
ASA 2.0000 0.1405 0.196
ATA 2.0000 0.7759 0.177 *
Session 4: More complex models
¡
Make sure no ME
§
Are all zeros really zeros?
¡
In principle valid for sampled data (admissible)
¡
MNAR impossible to check (but robustness can be assessed)
§
Are missing data “different” than observed?
¡
If attributes are missing we can use a similar
technique of data-‐augmentation (not in Pnet
yet)
Unobserved data: snowball sampling
Unobserved data: snowball sampling
Unobserved data: snowball sampling
Unobserved data: snowball sampling
Unobserved data: snowball sampling
Unobserved data: snowball sampling
Unobserved data: snowball sampling
missing data observed data
Sampling in/on networks
=
0
x =
- 1
1
Sampling in/on networks
=
0
0
1
x = 1
-
- 0
1
1 0
1
1
Sampling in/on networks
=
0
0
0
1 0
x = 1
-
- 0
1
1 0
1 0
0 1
Sampling in/on networks
=
0
0
0
1 0
1
0
0 0 0
0 0
0
0
x =
-
- 0
1
1 0
1 0
0 1
Sampling in/on networks
=
0
0
0
1 0
1
0
0 0 0
0 0
0
0
x =
-
-
- 0
1
0
1 0 - 0
1 0
- 1
0 1 1 0 -
Sampling in/on networks
=
0
0
0
1 0
1
0 0
?
0
? 0
? ?
? ? 0
0 0
0
x =
-
-
-
-
- 0
1
?
? 0
1 0 - 0 ? ?
1 0
?
? -
1
0 ?
0 1 1 - ?
Ignoring non-sampled?
=
0
0
0
1 0
1
0 0
?
0
? 0
? ?
? ? 0
0 0
0
x =
-
-
-
-
- 0
1
?
? 0
1 0 - 0 ? ?
1 0
?
? -
1
0 ?
0 1 1 - ?
What about alter – alter across ego?
=
0
0
0
1 0
1
0 0
?
0
? 0
? ?
? ? 0
0 0
0
x =
-
-
-
-
- 0
1
?
? 0
1 0 - 0 ? ?
1 0
?
? -
? 0 ?
0 1 ? - ?
Unobserved data: snowball sampling
Making some (brave) assumptions
(Handcock & Gile 2010)
we can fit an ERGM
(Wang et al. 2013)
to snowball sampled networks
• Importance sampling MCMCMLE (Handcock & Gile 2010)
• Stochastic approximation and the missing data principle (Orchard & Woodbury,1972) (Koskinen & Snijders, 2013)
• Bayesian data augmentation (Koskinen, Robins &
Pattison, 2010,2013) (
MPNet
)• Conditional MLE (Pattison, Robins, Snijders & Wang, 2013)(
SnowPNet
)Unobserved data: snowball sampling
Bayesian data augmentation (Koskinen, Robins & Pattison, 2010,2013) (MPNet)
• Need to know N
• Need to simulate un-observed ties
• Time-consuming
Conditional MLE (Pattison, Robins, Snijders
& Wang, 2013)(SnowPNet)
• No need to know N
• No need to simulate un-observed data
• … properties of conditional MLE unclear
Estimating ERGM for LARGE networks
Stivala et al. (2014)
Take many small snowball samples from your LARGE N network
Estimate Conditional MLE for each (Pattison, Robins, Snijders & Wang, 2013)
Pool estimates using Meta-analysis
techniques
Stivala et al. (2014)
Stivala et al. (2014)
Stivala et al. (2014)
Spatial embedding
Spatial embedding (Book Ch. 8)
306 actors in Victoria, Australia
Spatial embedding (Book Ch. 8)
306 actors in Victoria, Australia
... all living within 14 kilometres of each other
spatially embedded
Spatial embedding (Book Ch. 8)
306 actors in Victoria, Australia
... all living within 14 kilometres of each other
spatially embedded
Spatial embedding (Book Ch. 8)
Bernoulli conditional on distance Empirical probability
306 actors in Victoria, Australia
... all living within 14 kilometres of each other
Spatial embedding (Book Ch. 8)
Spatial interaction function: Tie probability as a function of distance
E.g. Attenuated Power-Law:
Pr(X
ij= 1 | d
ij) = p
1+ α d
ijγSpatial embedding (Book Ch. 8)
Spatial interaction function: Tie probability as a function of distance
The Attenuated Power-Law:
Pr(X
ij= 1 | d
ij) = p 1+ α d
ijγIs equivalent to:
Pr(X = x | D = (dij)) = exp{θ1 xij
i< j
∑
+θ2∑
i< j xij log(dij)}exp{θ1 uij
i< j
∑
+θ2∑
i< juij log(dij)}∑
u∈Xp = 1 α = e
−θ1γ = − θ
2with: AND: log(d
ij)
Spatial embedding (Book Ch. 8)
Edges -4.87* (0.13)
Alt. star
Alt. triangel
Log distance Age
heterophily -0.07* (0.01)
Gender
homophily -1.13* (0.61)
Spatial embedding (Book Ch. 8)
Edges -4.87* (0.13) 1.56* (0.65)
Alt. star
Alt. triangel
Log distance -0.78* (0.08)
Age
heterophily -0.07* (0.01) -0.07* (0.01)
Gender
homophily -1.13* (0.61) -1.13 (0.69)
Spatial embedding (Book Ch. 8)
Edges -4.87* (0.13) 1.56* (0.65) -4.79* (0.66)
Alt. star -0.86* (0.18)
Alt. triangel 2.74* (0.15)
Log distance -0.78* (0.08)
Age
heterophily -0.07* (0.01) -0.07* (0.01) 0.001 (0.07)
Gender
homophily -1.13* (0.61) -1.13 (0.69) 0.09 (0.83)
Spatial embedding (Book Ch. 8)
Edges -4.87* (0.13) 1.56* (0.65) -4.79* (0.66) -0.20 (0.87)
Alt. star -0.86* (0.18) -0.86* (0.2)
Alt. triangel 2.74* (0.15) 2.69* (0.14)
Log distance -0.78* (0.08) -0.56* (0.07)
Age
heterophily -0.07* (0.01) -0.07* (0.01) 0.001 (0.07) 0.002 (0.06)
Gender
homophily -1.13* (0.61) -1.13 (0.69) 0.09 (0.83) 0.07 (0.47)
ERGM: distance and endogenous dependence
explain different things
Bipartite and Multilevel ERGM
Multilevel
Level A Level B
Aij
Brs
Xir
The A-‐network
The X-‐network The B-‐network
Multilevel
¡
Network statistics can be derived based on the same dependence assumptions
¡
Different interpretation as we assume
dependencies between tie-‐variables of different types.
∑
⎭⎬⎫⎩⎨
⎧
+ +
+ +
= +
=
=
=
Q Q Q Q Q Q Q
Q Q Q
Q Q
Q
x b a z x
b z x
a z
x z b
z a
b z B x X
a
A ( , ) ( , ) ( , , )
) ( )
( )
exp ( )
( ) 1
, ,
Pr( θ θ θ
θ θ
θ θ
κ
Three network
variables A, B and X Within level effects Between level effects
Interaction between within level
and between level networks Cross level effects
Multilevel
¡
Bernoulli
¡
Markov
Affiliation based activity Affiliation based closure or homophily
¡
Social circuit and three-‐path
Affiliation assortativity Cross-‐level assortativity/entrainment
Multilevel
Multilevel: example, global fisheries
governance (Hollway & Koskinen, 2014)
Multilevel: example, global fisheries
governance (Hollway & Koskinen, 2014)
IRQ
HTI
ERI
CPV TGO
GUY
BEN SLE LBR
SOM GIN
KWT
GRD GHA
BEL
SDN
LBY VUT LCA
FSM
FJI
GMB BRB
AGO CIV LBN
PRK
MDG
SLB
DMA NGA
IRL
TUV
DJI
QAT
MUS DZA
MMR
PNG
URY
CMR MLT
BHS
KHM ISR
GEO SYC
KNA
WSM CYP
VCT
MHL SYR
SVNBHR
ATG BIH
BGD
MNE LKA
ARG
TUN CHNCAN OMN
MAR
BRN
SGP
IND ALBVNM IRN YEM
PAK HND
IDN
MYS
NOR
TUR
PHL ARE GNQ GRC
TTO
PER SLV PRT
BGR
DNK
MOZ
HRV SAU
CHL ESP ROK
ISL PAN
POL
JAM
THA COD
CUB UKR
FIN PLW JPNSWE
GBR ZAF
TLS
LVA
GAB
NAM
EGY
TON TZA KEN LTU RUS NZL
BLZ
CRI
SEN GTM
VEN
COL
BRA MEX
ITA FRA
NLD
KIR SUR
EST AUS USA JOR
DOM
MRT
COG ROU
NIC DEU
GNB
ECU MCO BOL
PRY
SMR LIE TCD
LSO
MNG
AND
LUX BLR
BWA CZE
CAF
ZWE MLI
KGZ BTN
ARM
NER BFA
SWZ
AFG TJK
SVK
COM UZB
NPL HUN RWA
MDA CHE
AZE YUG
AUT
TKM
STP
ETH KAZ
MKD MDV
BDI ZMB
LAO UGA
MWI TWN
NRU
EU
IRQ
HTI ERI
CPV TGO
GUY
BEN
SLE
LBR GIN
SOM KWT GHA
GRD
BEL SDN
LBY VUT
LCA
FSM
FJI
GMB BRB
AGO
CIV LBN
PRK MDG
SLB
DMA
NGA IRL
DJI TUV QAT
MUS
DZA MMR
PNG CMRURY
MLT BHS
KHM ISR
GEO
SYC KNA
WSM CYP
VCT
MHL
SYR BHR
SVN
ATG BIH
BGD
MNE
LKA
ARG
TUN
CAN CHN
OMN
MAR BRN
SGP
ALB
IND VNM
IRN PAK YEM
HND
IDN MYS
NOR TUR
PHL
ARE
GNQ
GRC
TTO
PER
PRT SLV
BGR
DNK MOZ HRV
SAU
ESP CHL ROK
ISL PAN
POL JAM
THA COD
CUB UKR
FIN
PLW SWE
JPN GBR ZAF
TLS
LVA
GAB NAM
EGY TON
TZA KEN
LTU NZL RUS
BLZ CRI
SEN
GTM VEN
COL
BRA
MEX
ITA FRA
NLD
KIR SUR
EST AUS
USA JOR
DOM
MRT
COG
NIC ROU
DEU
GNB ECU MCO
BOL
PRY SMR
LIE
TCD
LSO
MNG
AND
LUX
BLR BWA
CZE
CAF
ZWE
MLI KGZ
BTN
ARM NER
BFA
SWZ AFG
TJK SVK
COM
UZB NPL
HUN
RWA
CHE MDA
AZE
YUG AUT
TKM
STP ETH
KAZ
MKD
MDV
BDI ZMB
LAO
UGA
MWI TWN
NRU
154601
156815
153519
144077 153515
001378
001814 153423
001362
001384 000753 001400
001924
001383
001333 001880
153407
001817
001826 153419
153399 001657
153861
001237
154133 001199
001239 001217
001390
001183
001227 153638
153759 001858
001835 001889 001381
001251
001173
001911
153163
001958
001964
001232 154445 001388
001127 153563
001978
001132 001133
002009
001040 001991 001056
001043
153358 001112
153833
000976 001389
002034
001101 000997
153294
153889 000826
000871 000830
153354 000767
000732 002025
000741 000848
000134
000735
000536
000103 000101 000535
153475 000133
000480 000097
000587
000431 000479
000096
000502 000433
000426 000077
153479
000581 000425
000423 000457
000476 000076
000422 000141
000143
000144 000510
153179 153175
153191
000475 000128
000092
000560 144643
000730 153251 153896
000072
The A-‐network
The X-‐network
The B-‐network
Multilevel: example, global fisheries
governance (Hollway & Koskinen, 2014)
The B-‐network
global fish. (Hollway & Koskinen, 2014)
Effects Parameter Stderr t-‐ra/o SACF
EdgeA -‐2.0222 9.526 -‐0.02 -‐0.005
ASA 0.3048 0.136 -‐0.015 -‐0.01 *
ATA 0.3388 0.113 0.007 0.001 *
GDP_SumA 0.1681 0.865 -‐0.021 -‐0.009
GDP_ProductA -‐0.0016 0.078 -‐0.022 -‐0.012
species_SumA 0.0084 0.002 -‐0.011 -‐0.02 *
distance_EdgeA -‐1.0614 0.095 -‐0.019 -‐0.01 *
XEdge -‐9.1181 0.896 -‐0.061 0.061 *
IsolatesA -‐6.0824 0.748 0.018 -‐0.014 *
XASA 4.3784 0.32 -‐0.063 0.055 *
XASB -‐1.4607 0.396 -‐0.061 0.062 *
XACA -‐0.4665 0.03 -‐0.067 -‐0.014 *
XACB 0 0.02 -‐0.045 0.033
logGDPStateTreat_XEdge 0.1467 0.041 -‐0.07 0.063 *
Star2AX 0.0458 0.011 -‐0.008 -‐0.024 *
Star2BX -‐0.5834 0.084 -‐0.046 0.045 *
TriangleXBX 2.8928 0.197 -‐0.03 0.014 *
L3XBX 3.1433 1.267 0.057 0.007 *
ATXBX -‐0.0135 0.001 -‐0.031 -‐0.024 *
L3AXB -‐0.0036 0.011 -‐0.011 -‐0.014
Multilevel: example, global fisheries
governance (Hollway & Koskinen, 2014)
EdgeA 159 159.513 14.504 -‐0.035
Star2A 1145 1125.96 138.15 0.138
Star3A 7525 6986.738 621.013 0.867
Star4A 48877 44694.13 2383.26 1.755
Star5A 264331 245549.822 7529.189 2.494
TriangleA 37 38.189 11.016 -‐0.108
ASA 326.9238 328.5935 42.319 -‐0.039
ASA2 326.9238 328.5935 42.319 -‐0.039
ATA 85.9063 86.5708 19.376 -‐0.034
A2PA 1062.2734 1045.7699 110.027 0.15
AETA 183.082 197.2195 62.753 -‐0.225
coast_SumA 18227.8 17848.9168 1698.574 0.223
coast_DifferenceA 5174.2 6028.7824 655.418 -‐1.304
coast_ProductA 552268.66 507412.9956 57057.678 0.786
GDP_SumA 3581.3 3592.8811 325.357 -‐0.036
GDP_DifferenceA 248.9 248.4494 19.151 0.024
GDP_ProductA 20108.9029 20174.487 1831.761 -‐0.036
species_SumA 11426 11468.939 1380.074 -‐0.031
species_DifferenceA 5458 6207.877 761.398 -‐0.985
species_ProductA 229363 220670.215 43495.224 0.2
distance_EdgeA 1245.6313 1249.2669 112.117 -‐0.032
XEdge 1744 1744.314 25.393 -‐0.012
XStar2A 12297 11666.135 219.44 2.875
XStar2B 41955 41470.162 476.587 1.017
XStar3A 80932 74966.58 1091.676 5.464
XStar3B 1366512 1354528.805 5253.258 2.281
X3Path 941997 937576.735 23198.243 0.191
X4Cycle 75039 69696.854 1899.436 2.812
XECA 2265537 2120319.446 72782.425 1.995
XECB 10386840 10114999.01 223227.681 1.218
Multilevel: example, global fisheries
governance (Hollway & Koskinen, 2014)
IsolatesA 6 6.008 1.966 -‐0.004
IsolatesB 0 0.193 0.44 -‐0.438
XASA 2785.5996 2786.1502 48.835 -‐0.011
XASB 3047.4525 3048.0564 50.356 -‐0.012
XACA 4344.3993 4345.9997 60.884 -‐0.026
XACB 22625.6352 22624.8278 104.667 0.008
XAECA 293489.2989 276096.6764 7695.013 2.26
XAECB 298680.7706 278382.2566 7602.204 2.67
logGDPStateTreat_XE
dge 18782.09 18786.8598 267.542 -‐0.018
Star2AX 5261 5272.267 529.509 -‐0.021
StarAA1X 6539.459 6834.3932 922.611 -‐0.32
StarAX1A 9277.9278 9303.4066 956.331 -‐0.027
StarAXAA 3410.2835 3409.8379 78.967 0.006
TriangleXAX 1071 888.225 107.855 1.695
L3XAX 271.8552 253.8131 25.128 0.718
ATXAX 47884 45349.46 5516.119 0.459
EXTA 2273 2638.183 853.083 -‐0.428
Star2BX 2030 2030.261 26.211 -‐0.01
StarAB1X 1815.375 1824.7945 12.168 -‐0.774
StarAX1B 3833.0301 3833.2944 52.183 -‐0.005
StarAXAB 3160.9437 3161.8251 50.42 -‐0.017
TriangleXBX 396 396.673 16.435 -‐0.041
L3XBX 54.3754 54.38 0.832 -‐0.006
ATXBX 37668 37803.366 1808.02 -‐0.075
EXTB 552 566.508 5.745 -‐2.525
L3AXB 5383 5398.32 538.218 -‐0.028
C4AXB 1073 875.858 108.474 1.817
ASAXASB 8354.834 8659.1877 923.271 -‐0.33