Claude 3.5 Computer Use, developed by Anthropic, is an innovative AI system designed to automate desktop tasks in real-world environments. Unlike traditional software automation that relies on APIs or code-level access, Claude 3.5 operates through visual-only interactions, mimicking how a human navigates a computer interface. This study explores the model’s capabilities across domains like web navigation, office productivity, workflows, and video games, offering insights into its potential to revolutionize routine digital tasks while identifying the challenges that must be addressed for broader adoption.

 

This study evaluates Claude 3.5 Computer Use, and explores the model’s capabilities and limitations through tasks across domains like web navigation, office productivity, workflows, and video games. The study emphasizes the model’s Planning, Action, and Critic abilities while highlighting errors categorized as Planning Errors (PE), Action Errors (AE), and Critic Errors (CE). Additionally, the researchers present a universal Computer Use Out-of-the-Box framework to simplify GUI automation for researchers and users.

 

 

The following studied use-cases give an excellent cross section of potential uses for us to consider using in different capacities, and an evaluation of how a currently available model performed.  Note that while Claude cannot currently be used with protected information, there are still valuable uses.  (Also, what is available now in one model, will be available in others later.)

Use Case Summaries (Starting from Section 4.2 of the study)

4.2.1 Find ANC Headphones Under Budget $100 on Amazon

  • Planning: The model identified the user query, searched using keywords, and filtered results by price.
  • Action: It interacted with the search bar, typed the query, and clicked “Add to Cart.”
  • Critic: Confirmed the item was added by taking a screenshot and validating the action.

4.2.2 Browse Apple Official Site for Display with Accessories

  • Planning: Efficiently utilized the search function instead of navigating through complex menus.
  • Action: Interacted with text, buttons, and hyperlinks to locate and add items (e.g., accessories) to the cart.
  • Critic: Reflected on key steps, validated actions like adding AppleCare+ and the final cart content.

4.2.3 Fox Sports Subscription (Failed)

  • Planning: Attempted to find “Formula 1” in the menu but shifted erroneously to the “Account” tab.
  • Action: Correctly clicked the “MORE” button initially but failed to scroll further to locate “Formula 1.”
  • Critic: Misjudged task completion by prematurely assuming the “Account” section would lead to a solution.
  • Error: Planning Error (PE)—misinterpreted navigation flow.

4.3.1 Find Latest & Local Trending Music and Add to Playlist

  • Planning: Identified steps to locate music sections in Apple Music and add songs to a specific playlist.
  • Action: Navigated tabs, selected songs, and added them to the playlist by interacting with menus.
  • Critic: Validated each step via screenshots and ensured correct additions to the playlist.

4.3.2 Search for Products on Amazon and Record Prices in Excel

  • Planning: Combined web search and Excel data entry into a sequential plan.
  • Action: Performed searches, opened Excel, and accurately entered data.
  • Critic: Verified each stage (e.g., opening Excel, data entry) through screenshots and feedback.

4.3.3 Export and Download Online Document to Open Locally

  • Planning: Outlined steps to export a Google Spreadsheet and open it in Excel.
  • Action: Navigated menus, selected download options, and opened the file.
  • Critic: Verified each phase (e.g., download completion, file opening) through observations.

4.3.4 Install App from App Store and Report Storage Usage

  • Planning: Planned to locate “GoodNotes,” initiate installation, and check app size.
  • Action: Searched for the app, initiated installation, and verified app details.
  • Critic: Verified app size and installation process but noted required user authentication for completion.

4.4.1 Forward a Specific Email and CC Another Recipient

  • Planning: Detailed steps to locate the email, forward it, and add recipients.
  • Action: Opened the email, filled recipient fields, and clicked “Send.”
  • Critic: Verified all steps through screenshots, ensuring task completion.

4.4.2 Change Document Layout to A3 Size in Landscape Orientation

  • Planning: Outlined steps to adjust paper size and orientation in Word.
  • Action: Accessed layout settings and applied changes.
  • Critic: Confirmed successful updates through verification.

4.4.3 Two Columns Document

  • Planning: Planned to access the “Layout” tab and modify the column setting.
  • Action: Clicked relevant menus and applied two-column formatting.
  • Critic: Verified the document displayed in two columns.

4.4.4 Update Name and Phone Number on Resume Template (Failed)

  • Planning: Planned to replace specific text in a Word document.
  • Action: Only partially replaced the name and failed to select the full phone number.
  • Critic: Incorrectly confirmed task completion despite incomplete execution.
  • Error: Action Error (AE)—failed to execute precise text selection.

4.4.5 Gradient Fill Background

  • Planning: Planned to apply a gradient background using PowerPoint’s design tools.
  • Action: Initially failed to access the desired pane but adapted by switching to the Design tab.
  • Critic: Verified completion by observing the applied gradient fill.

4.4.6 Modify Slide Title and Draw a Triangle

  • Planning: Planned to add a title and insert a triangle shape.
  • Action: Typed the title, selected a triangle, and drew it on the slide.
  • Critic: Verified both title and shape additions.

4.4.7 Insert Numbering Symbol (Failed)

  • Planning: Planned to add numbering symbols to text.
  • Action: Applied bullet points instead of numbering and failed to adjust fully.
  • Critic: Incorrectly verified task completion despite errors.
  • Error: Critic Error (CE)—misjudged action success.

4.4.8 Find and Replacement in Worksheet

  • Planning: Planned to replace “$” with “SGD” in an Excel sheet.
  • Action: Used the “Find and Replace” dialog accurately.
  • Critic: Verified changes by counting replacements.

4.4.9 Insert a Sum Equation over Cells (Failed)

  • Planning: Planned to sum values in the “ACTUAL” column.
  • Action: Selected the wrong range (C6 instead of D6).
  • Critic: Failed to detect the incorrect range.
  • Error: Action Error (AE)—incorrect cell range selection.

4.5.1 Hearthstone: Create and Rename a New Deck for Battle

  • Planning: Planned to create a Mage deck and rename it.
  • Action: Selected deck options, renamed it, and finalized actions.
  • Critic: Verified changes at each step, ensuring the deck was ready.

4.5.2 Hearthstone: Hero Power

  • Planning: Evaluated game state and planned to use Hero Power.
  • Action: Activated Hero Power, targeted a minion, and ended the turn.
  • Critic: Verified improved board state post-action.

4.5.3 Honkai: Star Rail: Warp Automation

  • Planning: Planned to perform a 10-warp pull and skip animations.
  • Action: Executed warp, skipped animations, and reviewed the summary.
  • Critic: Verified successful pull and final outcomes.

4.5.4 Honkai: Star Rail: Daily Mission Clean-up Automation

  • Planning: Planned to complete a daily mission by automating specific steps.
  • Action: Set up and executed the mission, enabling auto-battle.
  • Critic: Monitored progress and verified task completion.