Process
How I design human-AI interfaces
The case studies on this site each describe one project. Read together, they describe one way of working. This page gathers it in one place, with a link back to the project each step came from.
At Microsoft I work at the intersection of AI and design, balancing what people need with the fast-advancing capabilities of AI. The steps below are not a framework I picked up. They are the moves that recur across the work, from Copilot in Word and Agent Mode back through the research and hardware projects that came before them.
Start where people already work
The capability goes into the tool the person already has open, not into a new one. Copilot in Word helps people draft, summarize, rewrite and edit documents with conversational agents, all within the familiar Word environment. pyReality put mixed reality data visualization directly inside Jupyter Notebooks so an analyst never had to switch platforms for modelling, rendering and viewing. My own MIDI controller plugged straight into Rekordbox rather than asking me to change how I perform.
Map the system before designing the interface
Before moving forward on the KeepTruckin developer platform, I investigated how it would affect the workflows that already existed in the App Marketplace. I mapped the current environment, drew high-level user flows for the new end-to-end journey from app creation and testing to publication, and split the project into phases so each change could be checked before the next. The same habit applies to an AI feature: know what the person does today before deciding what the agent should do for them.
From: Motive / KeepTruckin
Interview the people it is for, and keep them in the loop
On KeepTruckin I interviewed developers, identified their pain points and validated assumptions through regular feedback sessions. On Fidget Digits we ran semi-structured interviews with ten students before sketching anything, and feedback from users was taken at every step of the design process. The interviews are where the surprising constraints come from: those students preferred a handheld tracker to a wearable, which shaped the whole device.
Prototype how it feels, then iterate fast
For Agent Mode, the agentic loop in Word, Excel and PowerPoint, I prototyped interactions early to work out how autonomous capabilities would feel, iterated rapidly on concepts, and used what I found to help establish the direction. Fidget Digits went from sketches discussed with students to Rhino models to 3D prints that we could evaluate in the hand. The MIDI controller was tuned during rehearsals, adjusting mappings, response times and LED feedback. An agent has timing in it, so the prototype has to run, not just look right.
From: Agent Mode, Fidget Digits, MIDI Controller
Bring product and engineering in early
On KeepTruckin I collaborated closely with product managers and engineers from the start of the UX process so that technical feasibility and user needs stayed aligned. At User Evaluation I also took on a product management role, shaping the vision and requirements for transcription, AI insights and multimodal chat, and aligning each feature with the business objectives. Constraints found early are design material; constraints found late are rework.
Design the new pattern, not just the screen
Bringing transformer-based language models into Word meant designing new patterns and interactions, including tailored suggestions and proactive text transformations, rather than fitting AI into existing controls. Makerarm’s software did the same for fabrication: a smart tool detection system adjusted the interface to whatever head was attached, and a teaching mode let people train the arm by demonstration. When the system can act on its own, the pattern for how it proposes, shows its work and hands control back is the product.
From: Microsoft Copilot, Makerarm
Empower, not overpower
My design ethos is that technology should empower, not overpower, enhancing human experiences without overshadowing them. The expressive light lamp for shared study spaces was non-authoritative, non-invasive and subtle by design, and built so it never resembled a surveillance tool. Fidget Digits was meant to make students more self-aware without distracting them. For an agent, this is the line between a feature people keep and one they switch off.
From: Current practice, Expressive light lamp, Fidget Digits
Evaluate in real use, revise, and share what you learn
Usability testing of the lamp showed it changed behaviour and also showed habituation, which pointed to the next iteration. Fidget Digits was revised after evaluation because the first prototype favoured function over ergonomics. On Agent Mode, sharing findings with the broader team informed key research decisions. And I use data visualization as a medium for presenting what I find: COVID in Pixels turned an NLP analysis of pandemic news into interactive charts so the patterns could be seen, not just reported.
From: Expressive light lamp, Fidget Digits, Agent Mode, COVID in Pixels
Where this shows up
Each link opens that case study on the canvas.
- Microsoft Copilot
Agentic drafting, summarizing, rewriting and editing in Word.
- Agent Mode
Early design exploration of the agentic loop in Word, Excel and PowerPoint.
- User Evaluation
An AI-native platform for customer understanding at scale.
- Motive / KeepTruckin
A self-service developer platform for fleet management integrations.
- Makerarm
A browser-based control interface for a desktop fabrication system.
- Fidget Digits
A self-tracking device for fidgeting during online lectures.
- pyReality
Mixed reality data visualization inside Jupyter Notebooks.
- Expressive light lamp
A subtle physical-distancing aid for shared student spaces.
- COVID in Pixels
NLP analysis and visualization of humanitarian news during the pandemic.
- MIDI Controller
A custom Arduino controller for three-deck mixing.
Further reading
I write about agentic UX in the writing section, drawing on the same projects.