COMP90073 · Trustworthy Machine Learning · Workshop Week 3

Privacy leaks through the interface

Nobody hands the attacker the dataset.
They hand over answers, a model, or an anonymised table — and that is enough.

QR code linking to hesamasad.github.io/week-3-privacy-workshop
Scan to run the labs on your own device hesamasad.github.io/week-3-privacy-workshop

← → or space to navigate · S facilitator notes · O overview · F fullscreen

One workshop, three things you might release

PART 01 · Q1–Q2

A QUERY INTERFACE

Combine accurate answers until one person's value falls out of the arithmetic.

differencing · repeated noise
PART 02 · Q3–Q4

A TRAINED MODEL

Read membership off the model's confidence, then compose that into a feature guess.

membership · feature inference
PART 03 · Q5

AN ANONYMISED TABLE

Hide which row is Alice — and still hand over what Alice is sick with.

k-anonymity · homogeneity
The habit we are building: before you believe any attack number, write down the adversary's access, their auxiliary knowledge, and the strongest baseline that uses no attack at all.

The shared threat model

D Sensitive data people · attributes · labels
release→
f Release mechanism queries · model · table
observe & combine→
? Adversary auxiliary knowledge + strategy

THE UTILITY QUESTION

What must the release preserve to be worth publishing at all?

THE PRIVACY QUESTION

What do many observations reveal that any one of them does not?
1

The query interface

Every answer is a large, safe-looking aggregate. The leak lives in the differences between them.

Worksheet questions 1 & 2 · Lecture 3a
QUESTION 1 · DIFFERENCING ATTACK

Can suppressing small counts protect Alice's income?

You know: Alice is 29, and she is the only 29-year-old in the database.

The interface: \(\operatorname{count}(\text{age} \in [a_1,a_2],\ \text{income} \in [i_1,i_2])\). Any count of \(s\) or fewer is suppressed.

Your goal: learn Alice's income bracket without a single query that returns her alone.

THE OBVIOUS QUERY

\(\text{age} \in [29,29]\)
suppressed

SO ASK “EVERYONE EXCEPT 29”

\([0,28]\) and \([30,120]\)
two ranges, both answered

AND SUBTRACT FROM THE TOTAL

\(\text{all}-\text{young}-\text{old}\)
= 1 iff Alice qualifies
Note what this costs: three large queries. Each is individually harmless. Only their combination names a person.

Lab 1 · Break the suppression yourself

Compose real range queries. The interface refuses anything small. The bar shows what you have proved about Alice so far — nothing is drawn in advance.

Why suppression cannot see this coming

\[ \begin{aligned} &\underbrace{C([0,120],\,\text{inc}\le i)}_{\text{large}} -\underbrace{C([0,28],\,\text{inc}\le i)}_{\text{large}} -\underbrace{C([30,120],\,\text{inc}\le i)}_{\text{large}} \\[4pt] &\qquad=\;\mathbf{1}\!\left[\text{Alice's income}\le i\right] \end{aligned} \]

THE DEFENCE

Checks each answer against a threshold. Every answer here passes.

THE ATTACK

Uses a combination of answers. No individual answer is ever small.
Worksheet edge case: if Alice's income is among the \(s\) smallest, the counts for \(\text{income} \le i\) are themselves too small to release. Switch the band to \(\text{income} \ge i\) and walk \(i\) downward — those counts are large. Press Try the low-income case in the lab to watch it happen.

Checkpoint · what actually made the leak possible?

QUESTION 2 · FRESH NOISE PER QUERY

Suppression is replaced by noise in \(\{-1, 0, 1\}\)

WHAT CHANGED

Small counts are no longer withheld. Every answer — including the direct one about Alice — comes back, with an independent \(\eta \in \{-1,0,1\}\) drawn fresh each time you ask. The number of queries is unlimited.

So the query that Question 1 could not use is now available.

WHAT THE ATTACKER ASKS

\(r=\operatorname{count}(\text{age}=29,\ \text{income}=v)+\eta\)

Alice is the only 29-year-old, so the true count is \(1\) when \(v\) is her income and \(0\) otherwise.

Predict before you test: the noise is zero-mean and bounded. Which answers could you see in one case that you could never see in the other?

Lab 2 · Ask the same question a thousand times

Two routes: a single answer at either extreme of the noise range, or a running mean whose 99% interval clears 0.5.

Lab 3 · The same attack with no interface at all

A published table of aggregates about three people. Release one statistic at a time and watch the set of databases still consistent with it collapse.

So how much noise is enough?

Dinur–Nissim: with \(2^{n}\) subset queries and noise bounded by \(E\), an adversary reconstructs all but \(4E\) entries.

TOO LITTLE NOISE

Answers must stay consistent with each other. Wrong databases get ruled out until only the truth survives.

ENOUGH NOISE

\(E = O(\alpha\sqrt{n})\) leaves \(O(\alpha^{2}n)\) entries unrecovered — but degrades every legitimate answer too.

TOO MUCH NOISE

The release stops being worth publishing. This is the privacy–utility trade-off, priced.
The lecture's one-line summary: giving overly accurate answers to too many questions will inevitably destroy privacy — and "a little random noise" is not a defence until you have said how little, and how many.
2

The trained model

Nothing is published but weights and predictions — and the model still remembers who taught it.

Worksheet questions 3 & 4 · Lecture 3b
QUESTION 3 · WHAT DOES “80% SUCCESS” MEAN?

A friend reports an 80% attack. Should you be impressed?

Their attack predicts whether someone earns 80K+. The population has prevalence \(p\). Drag \(p\) and watch what a strategy with no model and no data already achieves — then check whether the answer survives a stricter reading of “success rate”.

Membership inference is a two-world game

WORLD 1
\(D + z\)→ train →\(f_1\)

The target record \(z\) was in the training set.

?
WORLD 0
\(D\)→ train →\(f_0\)

The target record \(z\) was not.

WHY IT COUNTS AS A HARM

Membership in a study of a disease, or of substance-abuse treatment, is the sensitive fact — even if the record is never recovered.

WHAT SUCCESS MEANS

Beating chance is a lower bound on privacy loss. A failed attack proves nothing about privacy.

Two ways to actually play it

01Shadow training sets Draw data resembling the target's, and train k shadow models on it through the same API.
02Label the outputs Query each shadow model on its own members and on held-out records. Tag every prediction “in” or “out”.
03Train the attack model A classifier that maps (prediction vector, class label) to in/out. You know the labels because you built the shadows.
04Or just threshold Salem et al.: take the highest posterior and compare it to a threshold \(T\). No shadow models needed.
Both routes rest on one assumption: the model is more confident on data it was trained on. Next lab: make that assumption true or false and watch the attack follow.

Lab 4 · Overfitting is the dial, the threshold is just the readout

The train–test gap moves the member scores; \(T\) decides where you cut. Watch what happens to the attack when the gap goes to zero.

QUESTION 4 · COMPOSING TWO ATTACKS

Turning a membership oracle into a feature guess

WHAT YOU HOLD

age = 44
postcode = 3053
occupation = nurse
income = ?

You also know this person's record is in the training set.

THE RECIPE

  1. Enumerate the missing feature's domain.
  2. Build one completed record per candidate value.
  3. Run the membership attack on each of them.
  4. Guess the value whose record looks most like a member.
Before you run it: only one completed record is real, so only one is genuinely a member. What is the success rate you must beat for this to mean anything?

Lab 5 · Does the composition actually beat guessing?

One record shown live; the metrics average 400 simulated records. Widen the domain, or take the overfitting away, and see what survives.

What makes a model leak — and what helps

RISK FACTORS

  • Overfitting — poor generalisation means memorisation.
  • Many classes, sparse ones — small groups are hit hardest.
  • Outliers — atypical records are the most exposed.
  • Model capacity — more parameters, more room to memorise.

PARTIAL DEFENCES

  • Regularisation, dropout — reduce the gap you saw in Lab 4.
  • Hide confidence scores — but label-only attacks still work.
  • Generative models memorise too: GPT-2 emitted verbatim contact details from its training data.
Carry this into Part 3: attacks are audits. They can demonstrate a failure; they can never certify success.
3

The anonymised table

We stop hiding whether you are in the data, and start hiding which row is you. That is a different promise.

Worksheet question 5 · Lecture 4

Removing names is not anonymisation

THE MASSACHUSETTS GOVERNOR

Medical records were released with names and SSNs stripped. The voter roll was public. They overlap on postcode, birth date, sex — enough to re-identify the governor's own record.

87% of the US population is uniquely identified by that triple alone.

THREE KINDS OF COLUMN

Key identifiers — name, address, phone. Always removed.

Quasi-identifiers — age, postcode, sex. Individually harmless; jointly identifying.

Sensitive attributes — disease, income. The thing we are protecting.

And the set grows: if \(Q\) is a quasi-identifier set, so is \(Q \cup \{A\}\) for any other attribute. Whatever the adversary can look up becomes a quasi-identifier.

The definition is about equivalence classes

\[ D\text{ is }k\text{-anonymous w.r.t. }Q \iff \min_{r\in D}\;\bigl|\{r' \in D : r'[Q]=r[Q]\}\bigr| \ge k \]

GENERALISE

Replace values with coarser ones: 53715 → 5371* → 537**

SUPPRESS

Drop the outlying records that refuse to join a class.

MONOTONICITY

Coarser is never less anonymous — so search the lattice upward and stop at the first success.
Read the definition carefully: it constrains class size. It says nothing whatsoever about the sensitive column.

Lab 6 · Generalise until every class is big enough

Set a target \(k\), then coarsen each quasi-identifier. Watch the class sizes, the information loss, and the last column.

QUESTION 5 · THE HOMOGENEITY ATTACK

Five indistinguishable people can still share one secret

Dataset: 60,000 distinct records. The sensitive attribute takes 3 values, uncorrelated with everything else.

Release: 5-anonymity with respect to all non-sensitive attributes.

Assume: the maximum possible number of groups, and treat groups independently.

(A)

How many groups can there be at most?

(B)

How many share one sensitive value, on average?

(C)

For how many people is the secret exposed?
Commit to numbers now. The next slide will mark them.

Lab 7 · Check your three answers

Type your predictions, press Check, then move \(k\) and \(v\) to see how fast the exposure shrinks — and what it costs.

What k-anonymity claims — and what it does not

FAILURE MODE

Homogeneity: a whole class shares the sensitive value.

FAILURE MODE

Complementary release: two k-anonymous tables intersect to something that is not.

FAILURE MODE

Sparsity: in high dimensions every record needs k close neighbours, so the table generalises into uselessness.

The five questions in one map

QWhat was releasedThe attackThe lesson
1suppressed countsdifference “all” against “all except 29”a per-answer rule cannot bound a transcript
2fresh noisy countsrepeat the direct query: watch for a 2, or averagezero-mean noise buys delay, not privacy
3an attack success ratecompare with \(p\) — or \(\max(p,\,1-p)\) if it meant accuracya number without a stated metric has no baseline
4a membership signalenumerate the feature domain, take the argmaxcompositions inherit their weakest stage — and still need a baseline
5a k-anonymous tablefind the homogeneous classesidentity ambiguity is not attribute privacy

Take-home principles

WHEN YOU READ AN ATTACK RESULT

  • State the access model and the auxiliary knowledge.
  • Ask which metric “success” means before choosing a baseline: \(p\) for precision, \(\max(p,\,1-p)\) for accuracy.
  • Ask over how many samples, and for a confidence interval.
  • Remember a failed attack certifies nothing.

WHEN YOU DESIGN A RELEASE

  • Analyse the whole transcript, and every linked release.
  • Keep membership, identity, and attribute privacy separate.
  • Price the noise: how much, for how many queries.
  • When the stakes justify it, demand a guarantee instead of a survived attack.
Where this goes next: differential privacy makes neighbouring datasets hard to distinguish whatever the adversary knows and however many times they ask — the two gaps that broke every mechanism today.

Sources

  • COMP90073 Lecture 3a — Aggregate Statistics and Reconstruction Attacks.
  • COMP90073 Lecture 3b — Privacy Attacks against Machine Learning Models.
  • COMP90073 Lecture 4 — K-Anonymity.
  • Workshop Week 3 — questions 1–5.
  • Dinur & Nissim (PODS 2003) · Garfinkel, Abowd & Martindale (CACM 2019) · Shokri et al. (S&P 2017) · Salem et al. (NDSS 2019) · Carlini et al. (USENIX Security 2021) · Samarati & Sweeney · LeFevre et al. (SIGMOD 2005).
Note: every number on these slides is simulated in your browser from synthetic data. Nothing is collected, and no real personal data is used anywhere in this deck.
End of Workshop Week 3

Can an answer that is safe on its own
stay safe once you have all the others?

Press O to revisit any lab · S for facilitator notes