The sequencing rule[SEQ]inAPRHL composes two approximate couplings while summing the approxima- tion parameters; this rule is a generalization of the standard composition theorem of differential privacy (Theorem4.1.4). In this section we extend theadvancedcomposition theorem of differential privacy, Theorem4.1.5, which allows trading off the"andδparameters when analyzing a composition of private mechanisms.
While the proof of the sequential composition theorem is fairly straightforward, the advanced compo- sition theorem follows from a more technical argument using Azuma’s inequality. It is not obvious how to extend the proof to approximate liftings, but fortunately we don’t need to. The key observation is that the "-distance condition on witnesses ensures differential privacy generalized to distributions overpairsof outputs. Therefore, we can directly generalize the advanced composition theorem to liftings by viewing the function mapping a pair of inputs to the left/right witness as itself differentially private.
However, there is an important catch: the advanced composition theorem assumes a symmetric
adjacency relation. In particular, the witnesses must satisfy a two-sided, symmetric distance bound to compose, but approximate lifting only gives a one-sided bound for witnesses. So, we first introduce a symmetric version of approximate lifting where the witnesses satisfy the bound in both directions. Then we develop an advanced composition theorem for symmetric liftings in two stages. First we prove an advanced composition theorem for"-distance, showing how to control the distance between the output distributions of two compositions if we can bound the symmetric distance between the output distributions of each step. Then, we give an advanced composition theorem given a symmetric approximate lifting at each step of a composition. To apply this principle inAPRHL, we introduce a symmetric judgment in APRHL and show how to prove it from standardAPRHL judgments, and we internalize advanced composition in a loop rule for symmetric judgments.
Remark5.4.1. The advanced composition theorem from differential privacy implicitly assumes that all mechanisms terminate with probability 1, so in this section we assume all commands are lossless; this is not a serious restriction as derivable judgments inAPRHL only relate lossless programs (Lemma4.3.3).
Remark5.4.2. While we focus on the advanced composition theorem, our technique provides a simple route to generalize other sequential composition theorems, like the optimal composition theorem and the heterogeneous composition theorem (Kairouz, Oh, and Viswanath,2017), and composition theorems where the parameters can be selected adaptively (Rogers, Vadhan, Roth, and Ullman,2016).
Symmetric approximate liftings
We first introduce a symmetric version of approximate lifting.
Definition 5.4.3. Letµ1,µ2be sub-distributions overA1andA2, and letR⊆A1×A2be a relation. Let ?be an element disjoint fromA1andA2. Two sub-distributionsµL,µRover pairsA?1×A?2arewitnesses
for thesymmetric(",δ)-approximateR-liftingof(µ1,µ2)if:
1. π1(µL) =µ1andπ2(µR) =µ2; 2. supp(µL)∪supp(µR)⊆R?; and 3. d"(µL,µR)≤δandd"(µR,µL)≤δ.
(RecallS?is the setS∪ {?}, andR? is the relationR∪(A1× {?})∪({?} ×A2).) When the particular witnesses are not important, we sayµ1andµ2are related by thesymmetric(",δ)-lifting ofR, denoted
µ1R
](",δ)
µ2.
This definition is nearly identical to standard approximate liftings (Definition4.2.2) except it requires the distance bound in both directions. The two-sided bound in a symmetric lifting implies two standard approximate liftings: ifµ1R](",δ)µ2holds, thenµ1R](",δ)µ
2andµ2(R−1)
](",δ)µ
1both hold by taking
witnesses(µL,µR)and(µ>
R,µ>L)respectively, sinced" µ>R,µ>L
=d"(µR,µL). In general, the converse may not be true. However when the relationRis of a particular form, we can construct a symmetric approximate lifting by giving two approximate liftings.
Lemma 5.4.4. Suppose S1,S2 are subsets of A1,A2 respectively, and we have maps f1 : A1 →B and f2:A2→B. Define a relationRonA1×A2by
a1Ra2 ⇐⇒ a1∈S1∧a2∈S2∧f1(a1) =f2(a2).
Letµ1,µ2be sub-distributions overA1andA2. The approximate liftings µ1R](",δ)µ2 and µ2(R−1)
](",δ)
µ1, imply the symmetric approximate lifting
µ1R
](",δ)
µ2.
Proof. Let(µL,µR)witnessµ1R](",δ)µ2and let(νL,νR)witnessµ2(R−1)](",δ)µ1. For everyb∈B, define subsets[b]A1¬f
−1
1 (b)⊆A1and[b]A2¬ f
−1
2 (b)⊆A2partitioningA1andA2. First, we have µ1([b]A 1) =µL([b]A1×A ? 2) ≤exp(")·µR([b]A1×A ? 2) +δ =exp(")·µR([b]A1×[b]A2) +δ ≤exp(")·µR(A?1×[b]A2) +δ =exp(")·µ2([b]A2) +δ.
Define non-negative constants:
ρ(b)¬max(µ1([b]A
1)−exp(")·µ2([b]A2), 0).
Then
µ1([b]A1)≤exp(")·µ2([b]A2) +ρ(b),
with equality ifρ(b)>0. It is not hard to showPb∈Bρ(b)≤δ; letB0¬{b∈B|ρ(b)>0}. Then
µ1(∪b∈B0[b]A1) =exp(")·µ2(∪b∈B0[b]A2) +
X
b∈B0 ρ(b), but Theorem5.1.2bounds the left side:
µ1(∪b∈B0[b]A1)≤exp(")·µ2(∪b∈B0[b]A2) +δ.
By a similar calculation with(νL,νR)in place of(µL,µR), we have a symmetric boundµ2([b]A2)≤ exp(")·µ1([b]A1) +σ(b)for minimal non-negative constantsσ(b)such that
P
b∈Bσ(b)≤δ. Note that ρ(b)andσ(b)can’t both be strictly positive, by minimality. We define witnesses
ηL(a1,a2)¬ µ1(a1)·µ2(a2) µ2([b]A2) 1−µ1([ρ(bb]A)1) :f1(a1) =f2(a2) =b µ1(a1)·ρ(b) µ1([b]A1) : a2=? 0 : otherwise.
ηR(a1,a2)¬ µ1(a1)·µ2(a2) µ1([b]A1) 1−µ2([σ(bb]A)2) :f1(a1) =f2(a2) =b µ2(a2)·σ(b) µ2([b]A2) : a1=? 0 : otherwise.
Throughout, if a denominator is 0 we take the fraction to be 0 as well. Since supp(µ1) ⊆ S1 and supp(µ2)⊆ S2 by the marginal and support conditions of the two asymmetric liftings, supp(ηL)and supp(ηR)are contained inR?.
For the first marginalπ1(ηL)(a1), ifµ1([f1(a1)]A1)is zero thenρ(f1(a1)) =0 by minimality and µ1(a1) = 0, so ηL(a1,a2) = 0 for all a2 ∈ A2. Otherwise ifµ2([f1(a1)]A2) = 0 then ρ(f1(a1)) = µ1([f1(a1)]A1)by minimality, andηL(a1,a2) =µ1(a1)fora2=?and zero fora2∈A2. By a symmetric
argument, the second marginal is similar.
To check the symmetric distance conditions, take any setW⊆A?1×A?2. We want to compare
ηL(W) = X (a1,a2)∈W0 ηL(a1,a2) + X (a1,?)∈W ηL(a1,?) with ηR(W) = X (a1,a2)∈W0 ηR(a1,a2) + X (?,a2)∈W ηR(?,a2),
whereW0¬W∩(A1×A2). We claim (i)ηL(a1,a2)≤exp(")·ηR(a1,a2)for all(a1,a2)∈A1×A2, and
(ii)P(a
1,?)∈WηL(a1,?)≤δ. Without loss of generality, we assumeWis contained inR
?.
To show (i), let b¬ f1(a1) = f2(a2). If eitherµ1([b]A1)orµ2([b]A2)are zero then the relevant
probabilities inηLandηRare zero as well. Otherwise there are three cases. If bothρ(b)andσ(b)are both zero, then
ηL(a1,a2) ηR(a1,a2)= µ1([b]A 1) µ2([b]A2) ≤exp(").
Ifρ(b)>0, thenσ(b) =0 andµ1([b]A1)>0. Ifµ2([b]A2) =0 then the claim is immediate; otherwise,
ηL(a1,a2) ηR(a1,a2)= µ1([b]A1) µ2([b]A2) 1− ρ(b) µ1([b]A1) = µ1([b]A1)−ρ(b) µ2([b]A2) =exp(")
where the final equality is by minimality ofρ(b). Similarly ifσ(b)>0, thenρ(b) =0 andµ2([b]A2)>0
so ηL(a1,a2) ηR(a1,a2)= µ1([b]A1) µ2([b]A2) µ2([b] A2) µ2([b]A2)−σ(b) = µ1([b]A1) µ2([b]A2)−σ(b) = µ1([b]A1) exp(")·µ1([b]A1) ≤exp("), where the final equality is by minimality ofσ(b); note that ifµ2([b]A2) =σ(b), thenµ1([b]A1) =0, ηL(a1,a2), andηR(a1,a2)are all zero. This establishes (i).
Showing (ii) is more straightforward:
X (a1,?)∈W ηL(a1,?)≤ X a1∈A1 ηL(a1,?) =X b∈B ρ(b)≤δ. Hence we have ηL(W) = X (a1,a2)∈W0 ηL(a1,a2) + X (a1,?)∈W ηL(a1,?) ≤exp(") X (a1,a2)∈W0 ηR(a1,a2) +δ ≤exp(")·ηR(W) +δ,
giving the distance boundd"(ηL,ηR)≤δ. A similar calculation yields the symmetric boundd"(ηR,ηL)≤
Advanced composition of symmetric"-distance
Building up to advanced composition for symmetric approximate liftings, we first show advanced compo- sition for symmetric"-distance. Suppose we have two sequences ofnfunctions{fi}i∈[n],{gi}i∈[n]where
fi,gi:A→Distr(A)are such that for anya∈A, we can bound the"-distance between fi(a)andgi(a). Then we will bound the"-distance between the output distributions from then-fold compositions.
We use notation for the sequential composition of algorithms. Given a sequence of functions{hi}i∈[k]
wherehi:A→Distr(A), we writehk:A→Distr(A)for the composition of{hi}. Formally, we define hk(a)¬
¨
unit(a) :k=0
bind(hk−1(a),hk) :k>0.
(Recallunit:A→Distr(A)andbind:Distr(A)×(A→Distr(B))→Distr(B)are the monadic operations for distributions from Definition2.2.2.) We use the same notation for functions of typehi :D×A→ Distr(A), defininghk:D×A→Distr(A)as
hk(d,a)¬
¨
unit(a) :k=0
bind(hk−1(d,a),hk(d,−)) :k>0.
Proposition 5.4.5. Let fi,gi:A→Distr(A)satisfy d"(fi(a),gi(a))≤δand d"(gi(a),fi(a))≤δfor every i∈[n]and a∈A. For anyω∈(0, 1), let
"∗
¬"
Æ
2nln(1/ω) +n"(e"−1) and δ∗¬nδ+ω.
Then for every n∈Nand a∈A, we have d"∗(fn(a),gn(a))≤δ∗and d"∗(gn(a),fn(a))≤δ∗. Proof. LetBbe the booleans and definehi:B×A→Distr(A)as
hi(true,a)¬fi(a) and hi(false,a)¬gi(a)
for every a∈ A. Then d"(fi(a),gi(a)) ≤ δ and d"(gi(a),fi(a)) ≤ δimply hi(a,−) :B →Distr(A) is(",δ)-differentially private for everya∈A, where we viewBas the set of databases with the full
adjacency relation relating all pairs of booleans; in particular, this is a symmetric relation. Applying the advanced composition theorem of differential privacy (Theorem4.1.5),hn(−,a):B→Distr(A)is
("∗,δ∗)-differentially private for everya∈A. By Definition4.2.1we have
d"∗(hn(true,a),hn(false,a))≤δ∗ and d"∗(hn(false,a),hn(true,a))≤δ∗
for everya∈A. Sincehn(true,a) =fn(a)andhn(false,a) =gn(a)by definition, we conclude
d"∗(fn(a),gn(a))≤δ∗ and d"∗(gn(a),fn(a))≤δ∗.
Advanced composition of symmetric approximate liftings
Next, we extend Proposition5.4.5to symmetric approximate liftings; roughly speaking, we will apply the proposition to the functions mapping related inputs to the left or right witness distributions. We need a lemma about how witnesses are transformed under composition.
Lemma 5.4.6. Consider two sequences of functions{fi}i∈[n],{gi}i∈[n] with fi :A1→Distr(A1)and gi:
A2→Distr(A2), and a sequence of binary relations{Φi}i∈{0,...,n}onA1×A2.
Suppose we have two sequences of functions{li}i∈[n],{ri}i∈[n] with li,ri:A?1×A?2→Distr(A?1×A?2) producing witnesses to an approximate lifting ofΦi:
2. π1(li(a1,?)) =fi(a1)andπ2(ri(?,a2)) =gi(a2); and 3. supp(li(a1,a2))∪supp(ri(a1,a2))⊆Φ?i for(a1,a2)∈Φ?i−1
for every i∈[n]. Then lnand rngenerate witnesses for an approximate lifting relating the n-fold compositions: 1. π1(ln(a
1,a2)) =fn(a1)andπ2(rn(a1,a2)) =gn(a2)for(a1,a2)∈Φ0; 2. π1(ln(a1,?)) =fn(a1)andπ2(rn(?,a2)) =gn(a2); and
3. supp(ln(a1,a2))∪supp(rn(a1,a2))⊆Φ?
nfor every(a1,a2)∈Φ?0.
Proof. By induction onn. The base casen=0 is trivial. Whenn>0, the support condition follows by induction; the marginal conditions follow by a direct computation (LemmaA.1.1).
We are now ready to prove advanced composition for symmetric liftings.
Theorem 5.4.7. Letω∈(0, 1). Consider two sequences of functions{fi}i∈[n]and{gi}i∈[n]with fi:A1→ Distr(A1)and gi:A2→Distr(A2), and a sequence of binary relations{Φi}i∈[n]onA1×A2andΦ0⊆A1×A2. Suppose for every i∈[n]and(a1,a2)∈Φi−1, there is a symmetric approximate lifting:
fi(a1)Φi](",δ)gi(a2).
Then for every(a1,a2)∈Φ0, we have a symmetric lifting fn(a1)Φn
]("∗,δ∗) gn(a2) where"∗¬"p2nln(1/ω) +n"(e"−1)andδ∗¬nδ+ω.
Proof. For(a1,a2)∈Φi−1, let(µ( i)
L (a1,a2),µ( i)
R (a1,a2))witness the approximate lifting ofΦirelating fi(a1) andgi(a2). Define functions{li}i∈[n],{ri}i∈[n]of typeli,ri:A?1×A2?→Distr(A?1×A?2)as follows:
li(a1,a2)¬ µ(i) L (a1,a2) :(a1,a2)∈Φi−1 unit(?)×gi(a2) :a1=?,a26=? fi(a1)×unit(?) :a16=?,a2=? unit(?,?) :a1=a2=? ri(a1,a2)¬ µ(i) R (a1,a2) :(a1,a2)∈Φi−1 unit(?)×gi(a2) :a1=?,a26=? fi(a1)×unit(?) :a16=?,a2=? unit(?,?) :a1=a2=?
Given distributionsη1andη2overB1andB2respectively,η1×η2∈Distr(B1×B2)denotes the product
distribution defined in the expected way:
(η1×η2)(b1,b2)¬η1(b1)·η2(b2).
Now by assumption on(µ(Li)(a1,a2),µR(i)(a1,a2))and by definition whena1=?ora2=?, we have
d"(li(a1,a2),ri(a1,a2))≤δ and d"(ri(a1,a2),li(a1,a2))≤δ
for all(a1,a2)∈Φ?
i−1, and we have the marginal conditions required by Proposition5.4.5. Now take any (a1,a2)∈Φ0. By Proposition5.4.5, we have
d"∗(ln(a1,a2),rn(a1,a2))≤δ∗ and d"∗(rn(a1,a2),ln(a1,a2))≤δ∗.
Lemma5.4.6gives the marginal conditionsπ1(ln(a1,a2)) = fn(a1)andπ2(rn(a1,a2)) = gn(a2)and shows that supp(ln(a1,a2)), supp(rn(a1,a2))are contained inΦ?n, soln(a1,a2)andrn(a1,a2)witness the desired symmetric approximate lifting
fn(a1)Φn](" ∗,δ∗)
SYMINTRO Ψ¬e1〈1〉=e2〈2〉 ∧Ψ1〈1〉 ∧Ψ2〈2〉 `c1∼(",δ)c2:Φ=⇒Ψ `c2∼(",δ)c1:Φ−1=⇒Ψ−1 `c1≈(",δ)c2:Φ=⇒Ψ SYMELIM-L`c1≈(",δ)c2:Φ=⇒Ψ `c1∼(",δ)c2:Φ=⇒Ψ SYMELIM-R `c1≈(",δ)c2:Φ=⇒Ψ `c2∼(",δ)c1:Φ−1=⇒Ψ−1
Figure 5.5: Conversion rules between symmetric and standard judgments forAPRHL
WHILE-AC "∗ ¬" Æ 2Nln(1/ω) +N"(e"−1) δ∗¬Nδ+ω ω∈(0, 1) |=Φ→ev〈1〉 ≤0→ ¬e1〈1〉 |=Φ→e1〈1〉=e2〈2〉 ∀K∈N,`c1≈(",δ)c2:Φ∧e1〈1〉 ∧ev〈1〉=K=⇒Φ∧ev〈1〉<K `whilee1doc1≈("∗,δ∗)whilee2doc2:Φ∧ev〈1〉 ≤N=⇒Φ∧ ¬e1〈1〉
Figure 5.6: Advanced composition rule[WHILE-AC]forAPRHL
Symmetric judgments inAPRHL
In order to use advanced composition inAPRHL, we extend the logic with a new judgment modeling symmetric approximate liftings. We call such judgmentssymmetricjudgments.
Definition 5.4.8. A symmetricAPRHL judgment isvalidin logical contextρ, written ρ|=c1≈(",δ)c2:Φ=⇒Ψ,
if for any two inputs(m1,m2)∈¹Φºρthere exists an symmetric approximate lifting relating the outputs: ¹c1ºρm1¹Ψºρ
](¹"ºρ,¹δºρ)
¹c2ºρm2.
To prove these judgments, we extendAPRHL with a few proof rules. To keep our proof system as simple as possible, we introduce rules for symmetric judgments only where absolutely needed—namely, for advanced composition—and use the conversion rules in Fig.5.5to move between symmetric and standard, asymmetric judgments. The inverse relationΦ−1can be defined syntactically by simply interchanging the
tags〈1〉and〈2〉in a formulaΦ. Soundness of these rules is straightforward. Theorem 5.4.9. The rules[SYMINTRO],[SYMELIM-L], and[SYMELIM-R]are sound. Proof. Soundness of[SYMINTRO]follows by Lemma5.4.4. Soundness of[SYMELIM-L]and
[SYMELIM-R]follow by definition of symmetric approximate lifting.
An advanced composition rule forAPRHL
Finally, we internalize advanced composition of liftings as the loop rule[WHILE-AC]in Fig.5.6. Like the usual rule[WHILE], the guards must be synchronized and the loops run at mostN iterations. An
(",δ)-approximate coupling of the loop bodies gives an("∗,δ∗)-approximate coupling of the two loops, where"∗andδ∗are from the advanced composition theorem of differential privacy (Theorem4.1.5).
Theorem 5.4.10. The rule[WHILE-AC]is sound.
Proof. The proof follows essentially by Theorem5.4.7. As usual, we will leave the logical contextρ implicit. Consider two memories(m1,m2)∈¹Φ∧ev〈1〉 ≤Nºand two output distributions
µ1¬¹whilee1doc1ºm1 and µ2¬¹whilee2doc2ºm2.
We construct a symmetric approximate lifting relatingµ1andµ2. The value ofNis given by the logical contextρ; we treat it as a constant. We unroll the loopN times and define
µ0 1¬¹(ife1thenc1) N ºm1 and µ 0 2¬¹(ife2thenc2) N ºm2.
We claim¹e1ºm01=¹e2ºm02=falsefor allm01∈supp(µ01)and m02∈supp(µ02). We can use the valid symmetricAPRHL judgment in the premise and symmetric versions of the rules[SEQ]and[COND]to construct a symmetric approximate lifting
µ0 1Φ∧ev〈1〉 ≤0 ](N",Nδ) µ0 2. Since|=Φ∧ev〈1〉 ≤0→ ¬e1〈1〉, we have µ0 1¬e1〈1〉 ∧ ¬e2〈2〉 ](N",Nδ) µ0 2.
Letµ0L,µ0Rbe the corresponding witnesses. We knowπ1(µ0L) =µ01andπ2(µ0R) =µ02, and also supp(µ0L)∪supp(µ0R)⊆¹¬e1〈1〉 ∧ ¬e2〈2〉º,
so¹e1ºm01=¹e2ºm02=falsefor allm01,m02in the support ofµ0
1,µ02respectively. By the equivalences whilee1doc1≡(ife1thenc1)N;whilee1doc1
whilee2doc2≡(ife2thenc2)N;whilee2doc2, we know µ1=¹(ife1thenc1) N ºm1 and µ2=¹(ife2thenc2) N ºm2.
Defining a family of relations
Φi¬Φ∧(ev〈1〉 ≤N−i∨ ¬e1〈1〉),
we have
|=ife1thenc1≈(",δ)ife2thenc2:Φi=⇒Φi+1
for everyiusing the premise, sinceΦi ensures the guardse1ande2are equal in the initial memories.
By validity, for any pair of memories satisfyingΦi there is a symmetric approximate lifting of Φi+1
relating the two output distributions. We can apply Theorem5.4.7withA1=A2=State, functions
fi¬¹ife1thenc1ºandgi¬¹ife2thenc2º, and relationsΦito get the symmetric approximate lifting
µ1Φ∧(ev〈1〉 ≤0∨ ¬e1〈1〉) ]("∗,δ∗) µ2. Since|=Φ∧ev〈1〉 ≤0→ ¬e1〈1〉, we conclude µ1Φ∧ ¬e1〈1〉 ]("∗,δ∗) µ2
i←1; out←[]; whilei≤N∧ |out|<Cdo u←$ Lap "0(0); a←A−u; b←B+u; go←true; ans←(0, 0); whilei≤N∧godo v $ ←Lap"0/3(evalQ(i,d)); ifa<v<bthen noisy $ ←Lap"0(evalQ(i,d)); ans←(i,noisy);
out←ans::out;
go←false;
i←i+1
Figure 5.7: Between Thresholds
Remark5.4.11. Our approach narrowly limits the scope of symmetric judgments: they can be used in
[WHILE-AC]or eliminated to a standard judgment. There are at least two other choices. One option would be to define a full proof system based on symmetric judgments. Almost all the basic proof rules fromAPRHL would directly generalize, including the standard rules for program commands and the
Laplace rules. However, it is not clear how to generalize the more advanced rules, including[PW-EQ] and[UTB-L]/[UTB-R]. The optimal subset coupling (Theorem5.3.1) also does not directly generalize to symmetric liftings; this poses a problem for a symmetric version of[LAPINT].
For another option, we could avoid symmetric judgments entirely by fusing[SYMINTRO],[WHILE-AC], and[SYMELIM-L]together into a single rule. While this would suffice for our examples, it is conceptually clearer to separate symmetric and asymmetric judgments. Our design choice leaves room for other rules specific to symmetric approximate liftings, and clearly identifies the main bottleneck in converting from standard approximate liftings to symmetric liftings in the rule[SYMINTRO].