Microsoft Copilot can now ‘see’ what’s on your screen in Edge

Copilot Vision, Microsoft’s AI assistant feature that can interpret what’s on your screen and help you use apps, is now available for free use within the Edge browser, Mustafa Suleyman, CEO of Microsoft AI, announced on Bluesky today. Vision is a “talk-based experience,” as Microsoft calls it, meaning you use it by speaking into the air, then waiting for Copilot to respond.

Suleyman says if you opt into the feature, Copilot Vision can “literally see what you see on screen.” Suleyman suggests having Copilot Vision guide you through a recipe while you cook or having it “decode” job descriptions “and jump right into customized interview prep or cover letter brainstorming.” (Although it might not be the best idea to use AI for your resume.) According to a Microsoft support page, “Copilot Vision may highlight portions of the screen to help you find relevant information,” but it doesn’t actually click links or do anything on your behalf.

Broader, system-wide Copilot Vision features are still limited to Copilot Pro subscribers. With a subscription, Vision expands beyond Edge, letting you ask it to help you use features in Photoshop or video editing software, or guide you through a game like Minecraft, as it did for The Verge’s Tom Warren earlier this month.

To try out Copilot Vision, open this link to Microsoft’s website in the Edge browser. That should give you a prompt to opt into the feature, and once you’ve given permission, you can open the Copilot sidebar while on a website, click the microphone icon, and your Vision session begins, signified by a chime and your browser changing its hue. 

Or that’s how it should go. In my case, it took a couple of tries before Edge asked if I wanted to opt in. And once I could opt in and initiate a Vision session, the controls never appeared — as of this writing, I simply have a message floating over the bottom of my browser that says “One moment…” But I’m using a fairly old, underpowered laptop, so your mileage may vary.

According to Microsoft, the company logs Copilot’s responses to you but doesn’t collect your inputs, images, or page content while in a Copilot Vision session. When you’re ready to stop sharing your screen with Copilot, you can either end the session or close the browser window.