2020 to 2021Feedback shipped 2021 · framework followed
Impact
Framework adoptedPartner teams build their own recommendations
01The stakes
2020
Delivery is the machine learning system deciding where, when, and to whom an ad shows.
Relevance was hit or miss, and advertisers had learned to ignore recommendations that did not apply to them.
Accuracy wasn't the problem. Trust was.
Mid-flight recommendations helped advertisers improve results while campaigns were still running. In practice, three problems compounded: adding a new recommendation type was slow because there was no shared design and content framework and every detail needed its own approval; relevance was uneven, so adoption fell and trust eroded; and even good advice died in workflows too complex to act on.
Before · bespoke per case
Every recommendation designed one-off
No shared design or content framework
Relevance hit or miss for advertisers
Complex workflows blocking adoption
Approvals on every detail
After · unified framework
01One framework any team can apply
02Feedback from advertisers, in the product
03Simpler workflows to act on advice
Any team ships without starting from zero.
An ignored recommendation isn't neutral. It spends the trust the next one needs.
The core tension both workstreams answered
02The workstreams
2020 to 2021
Operated across Product Marketing, Content Strategy, UX Research, Engineering, Data Science, and the Interfaces Platform team.
Concept cards were built to provoke feedback rather than validate polish: advertisers needed projected impact before acting.
Two parallel workstreams. One designer connecting them.
A framework to solve the scaling bottleneck, and a feedback system to make the recommendations learn. The framework standardized what a recommendation looked like. The feedback system captured signal to improve which ones surfaced, and when.
Workflow types
Named five interaction patterns, Flow A through E: a recommendation surfacing in the account overview, in the campaign table, in an edit panel, in bulk edit, or edited in line. The names gave five teams one vocabulary.
Research
Ran concept studies built to provoke reactions rather than validate polish, asking what advertisers expected feedback to change, and whether a dismissed recommendation should ever return.
Framework
Built the design and content framework with the platform team, replacing per-case approvals with patterns any team could apply. Scaling the inventory stopped requiring scaling the process.
Feedback
Mapped lightweight advertiser feedback to signal the ranking system could read. The question the study surfaced that nobody had asked became a shipped feature: snooze.
Fig. 01 · Before the framework. Every delivery case got bespoke treatment: one-off design, one-off content, and approval on every detail. Cost scaled with the number of cases.
Every delivery case, rebuilt from zero
Lookalike drop-offBudget pacingSchedule gap+ every other case
→
One-off designOne-off contentManual approval
→
Shipsone at a time
Scaling the inventory meant scaling the process. The approval gate was the bottleneck.
Fig. 02 · Concept studies. Variations built to provoke reactions, not validate polish. The question they surfaced, whether a dismissed recommendation should ever come back, shipped as snooze.
Concept variations, built to provoke a reaction
Projected impact shown up frontThe reason a recommendation surfacedDismiss, with a reason attached
→
“Should a dismissed recommendation ever come back?”the question nobody had asked
→
Shipped as snooze
03The framework
2021
A dismissed recommendation could return with updated evidence and a “returning with new information” indicator: the cry-wolf problem, addressed directly.
Passive signals like adoption speed and time-to-action became additional ranking inputs.
Five workflow types, one framework. Any team could ship a recommendation without starting from zero.
The framework standardized how a recommendation looks, reads, and behaves across all five workflow types, then proved itself on a live case: Lookalike audiences (new people resembling existing customers). Feedback shipped in 2021: snooze, mute, and dismiss became signals the ranking system could finally read, the first closed loop between advertiser behavior and what the system showed next.
Fig. 03 · The workflow framework. One recommendation, specified across the five workflow types, Flow A through E, developed with the platform team.
One recommendation, five ways to act on it
A recommendationone framework: how it looks, reads, and behaves
↓
Flow ABottom sheet in Account Overview
Flow BBottom sheet in the campaign table
Flow CEdit-panel flow
Flow DBulk edit in the table
Flow EIn-line edit
Five teams, one vocabulary. No team started a recommendation from zero.
Fig. 04 · The drop-off case, live. A 2020 walkthrough of a real recommendation case: an audience drop-off surfaced mid-flight, with a guided path to act on it.
0:00
Fig. 05 · Resolution, automated. The same case through the simplified workflow: resolving the issue moves from a multi-step edit to a guided action.
0:00
Fig. 06 · Feedback expressions. Snooze, mute, and dismiss: advertiser feedback mapped to signal the ranking system could read. Shipped in 2021.
The first closed loop between behavior and what surfaces next
Recommendation shown
→
Advertiser respondsact · snooze · mute · dismiss
→
Mapped to signalplus adoption speed, time to action
→
Ranking learnswhich surface, and when
↻ a better next recommendation
Fig. 07 · Intent-aware feedback, in motion. The expressions respond to why a person is dismissing, so every dismissal finally carries a different signal.
0:00
5
Workflow types, one framework
↑
Resolution rate up, first live test
2021
Feedback expressions shipped
04What it outlasted
2021 onward
The framework’s job was other teams shipping without us. The outcomes are structural by design.
Partner teams began building their own recommendations on the framework, and the first live test moved the number that matters: resolution rate.
The feedback system gave ranking its first real signal: dismissals stopped looking identical. That groundwork became an engineering workstream on recommendation ranking, and the work kept compounding after I moved on.