Reimagining Independence: How Meta’s AI Models Are Helping the University of Pittsburgh Transform Assistive Robotics

Reimagining Independence: How Meta’s AI Models Are Helping the University of Pittsburgh Transform Assistive Robotics

Products AI Research Resources About AI Developers Try Meta AI

Open Source Reimagining Independence: How Meta’s AI Models Are Helping the University of Pittsburgh Transform Assistive Robotics July 27, 2026 • 8 minute read

In the heart of Pittsburgh, a quiet revolution is underway — one that promises to redefine what independence means for people with disabilities. At the center of this movement is the Human Engineering Research Laboratories (HERL) , a pioneering institute at the University of Pittsburgh, now leading an initiative with up to $41.5 million in funding from the Advanced Research Projects Agency for Health (ARPA-H) , a US research funding agency established to support transformative biomedical and health breakthroughs. With support from ARPA-H, the HERL team and ATDev , a pioneer in innovative assistive technology, are building the Robotic Assistive Mobility and Manipulation Platform Providing Independence for People with Disabilities (RAMMP), blending cutting-edge robotics, artificial intelligence, and user-centered design. For the estimated 5.5 million wheelchair users in the United States , preservation of mobility ensures individual opportunity and participation in everyday life. When assistive technology does not reflect the complexity of the real world, it can undermine confidence, independence, and physical safety. The statistics are sobering: over 100,000 wheelchair-related injuries are treated in US emergency departments each year , often due to trips and falls. As a result, the need for smarter, safer, and more adaptable technology is urgent.

RAMMP: A New Era for Assistive Mobility

The ARPA-H-supported Robotic Assistive Mobility and Manipulation Platform ( RAMMP ) project aims to address the shortcomings in current robotic mobility platform design through integration of advanced robotics, novel operating systems, and digital twin technology — creating a virtual simulation environment for safe, scalable testing and development. RAMMP’s approach also integrates artificial intelligence, including the use of several of Meta’s open source AI vision models, including DINO and Segment Anything Model (SAM) . DINO and SAM have already assisted in a variety of projects ranging from those focused on medical imaging to wildlife conservation. DINO, a self-supervised vision transformer, excels at learning visual representations from unlabeled data, making it ideal for environments where annotated datasets are scarce. SAM, on the other hand, is designed to “segment anything” and has the ability to identify and outline any object in an image or video with minimal prompting. Segment Anything’s versatility and accuracy have made it a foundational tool in domains where precise object recognition is paramount.

Engineering for the Real World: Running DINOv3 and SAM on Edge Devices

For people relying on assistive devices, every second counts. The unpredictable nature of everyday environments, such as a child darting across a sidewalk, the sudden appearance of a curb, or a dropped set of keys, requires an immediate reaction. Processing camera images and sensor data directly on the device, known as edge computing, empowers robotic mobility platforms to function as responsive tools. These tools must be robust and consistent across the wide range of dynamic environments in which people live. At the same time, deploying powerful AI models like DINOv3 and SAM on limited, battery-powered hardware presents significant engineering challenges. Real-world deployments must consider practical factors, including battery life, heat dissipation, unreliable network connectivity, and strict size and weight requirements. However, overcoming these constraints mean nothing to the end user if they can't perform their activities of daily living using these new robotic systems. These newer methods allow users to interact with the robot more naturally, using their immediate surroundings as context, freeing both engineers and users from having to design and navigate complex, time-consuming interfaces. The ability to use natural language combined with image data to query the user’s and robot’s environment and provide more direct commands directly reduces the cognitive load and amount of context switching required for a user to do something as simple as picking up a cup off a table. This functionality is already being integrated by the RAMMP team into their first prototype. Leveraging tools built off of DINO to enable querying the robot’s image sensors to detect automatic door buttons, cups, and curbs/ground for navigation assistance. With this functionality now ready for real-world testing, engineers are focusing on voice and touch input, letting users select and interact with specific objects in their surroundings. In addition to the existing challenges of ensuring accuracy and temporal coherence of the model outputs, this provides the additional challenge of ensuring robustness and predictability across user prompts and inputs. DINOv3 serves as a compact, efficient 'visual brain' for devices — a general-purpose foundation on which task-specific, lightweight modules can be layered for actions such as object detection or movement tracking, enabling reuse of visual data and conserving power. Applying both models as part of the development of robotics systems, engineers optimize models for edge devices, reducing memory footprint, using lower precision when appropriate, and deploying in formats tailored for real-world conditions. This ensures reliable, real-time operation for users. By running at practical resolutions and with efficient batching, both models stay fast and dependable, even on the compact, battery-powered hardware used in robotic mobility platforms and robotic arms, — sometimes trading a little bit of boundary precision and/or feature detail for the speed and stability needed by users on the go. This balance between precision and practicality is central to the project's philosophy. “For assistive robotics, performance is not measured by benchmark accuracy alone, but by whether a system can operate reliably in the unpredictability of everyday life," said Sivashankar Sivakanthan, Chief of Staff to the RAMMP project. “Running models like DINOv3 and SAM on-device is what enables real-time perception that users can trust - without relying on connectivity...