Adaptation: what robot foundation models actually learn?

Robot foundation models like NVIDIA's GR00T N1.7 look superficially like large language models: pretrain on broad data, specialize on narrow data. But the analogy breaks down in ways that matter for how we build, deploy, and maintain these systems. When we specialize an LLM, we are usually teaching it something it doesn't know. When we specialize a vision-language-action model for a new robot, we are usually teaching it nothing new at all. We are teaching it how to express what it already knows through a different body. This distinction is not pedantic. It changes the economics of industrial robotics, the architecture of deployment pipelines, and where competitive advantage will accrue.

Foundation models have arrived in robotics

Over the past three years, the machine learning community has internalized a particular pattern: pretrain a large model on internet-scale data, then adapt it cheaply for downstream applications. The pattern proved so productive for language that it was probably inevitable it would be exported to other domains. Robotics is the domain where the export is now happening in earnest, and where, we'd argue, the pattern needs the most careful reinterpretation.

NVIDIA's GR00T N1.7 is a useful concrete case. It is a 3-billion-parameter vision-language-action (VLA) model, released under a commercial license, that maps visual observations, natural language instructions, and robot proprioceptive state to continuous motor commands. Architecturally, it is a dual system. A vision-language model, a Cosmos-Reason2-2B backbone, handles high-level reasoning: parsing instructions, decomposing tasks, grounding language in what the cameras see. A separate diffusion transformer takes that high-level representation together with the robot's live state and denoises it into precise, real-time motor commands. NVIDIA describes these as System 2 and System 1, borrowing the deliberate-versus-reflexive framing from cognitive science, and the framing is apt: one component decides what to do, the other decides how the body should do it.

The inputs are considerably richer than anything an LLM consumes. A single inference step may condition on multiple RGB camera views, a language instruction, joint positions and velocities, end-effector pose, and gripper state. The outputs are not tokens but continuous action vectors whose dimensionality depends on the degrees of freedom of the particular robot. The model is, in effect, answering a question no text model has ever faced: given what I see, what I know about my body, and what I've been asked to accomplish, what sequence of physical motions should I execute over the next fraction of a second?

That question has two parts, and keeping them separate is the key to everything that follows. Understanding manipulation (how objects behave, how grasps succeed or fail, how tasks decompose into steps) is one kind of knowledge. Knowing how to move this particular body to act on that understanding is another. Pretraining supplies the first. Specialization mostly supplies the second.

What the pretraining actually teaches

A common misconception is that models like GR00T are trained primarily on robot demonstrations. The most interesting fact about GR00T N1.7's pretraining is how much of it isn't robot data at all.

The largest single ingredient is human data. N1.7 was pretrained on more than 20,000 hours of egocentric human video, NVIDIA's EgoScale corpus, spanning tasks from manufacturing and retail to healthcare and household work. This is first-person footage of humans assembling parts, opening cabinets, organizing shelves, manipulating tools, captured with ego cameras, wrist cameras, and hand tracking. None of it contains robot joint trajectories, because there is no robot in it. What it contains instead is something arguably more valuable and certainly more scalable: dense evidence about how manipulation works. Object affordances. Contact-rich interaction. Temporal structure. How multi-step tasks decompose. NVIDIA's underlying intuition is straightforward: humans and humanoid robots share the essentials of embodiment (two hands, a first-person viewpoint, a world full of graspable things), so sensorized human activity is a legitimate pretraining substrate for robot policies. Notably, NVIDIA reports a scaling relationship here: more human video data predictably improves downstream robot dexterity, without requiring that every behavior first be demonstrated on physical hardware through teleoperation. If that relationship holds up, it matters enormously, because human video is the one data source in robotics that scales the way text scaled for language models.

The second ingredient is robot demonstration data: corpora like Open X-Embodiment, DROID, and LIBERO, along with proprietary teleoperation across bimanual, semi-humanoid, and humanoid platforms. Each sample synchronizes camera images, joint states, end-effector poses, gripper state, executed actions, and a language instruction. This is where manipulation knowledge gets grounded into actual robot embodiments: where the model learns not just that a mug can be grasped by its handle, but what commanding a real arm through that grasp looks like as a trajectory.

The third ingredient is simulation. Physics simulators generate trajectories at a volume and diversity that physical data collection cannot approach: rare edge cases, safe failures, systematic domain randomization across lighting, textures, and object geometry. Simulation fills the gaps that would be prohibitively expensive, or unsafe, to fill with real hardware.

The composition of this mixture tells you what the resulting model is. It is not a library of robot-specific skills. It is a general model of manipulation, grounded across enough embodiments that "how bodies act on objects" has been learned as something closer to an abstraction than a lookup table. That is precisely why what happens after pretraining should not be described the way we describe LLM fine-tuning.

Where the LLM analogy breaks

Consider what happens when we fine-tune a language model on medical literature. The base model has broad general knowledge but shallow domain expertise; fine-tuning injects new semantic content. The model afterward knows things it did not know before: drug interactions, diagnostic criteria, clinical conventions. The input and output spaces are unchanged (text in, text out); what changed is the knowledge distribution.

Now consider adapting GR00T N1.7 to a new warehouse robot. The foundation model already understands grasping, object permanence, task sequencing, and visual grounding. That is exactly what tens of thousands of hours of pretraining bought. What it does not know is this robot: its kinematics, its joint limits, its gripper geometry, its controller's response characteristics, its camera placement, the peculiar lighting of this facility. Almost everything being learned during specialization concerns the interface between a stable body of physical knowledge and a novel embodiment and environment.

The two processes differ along nearly every axis that matters:

What changes. LLM fine-tuning primarily shifts semantic knowledge within a fixed input/output space. VLA adaptation primarily remaps input and output spaces (new sensor configurations, new action dimensionalities) around knowledge that stays largely fixed. GR00T's own machinery reflects this: the model indexes robots by embodiment tag, encodes each robot's proprioceptive state through embodiment-specific projections, and emits action vectors shaped to that robot's degrees of freedom. The scaffolding for "same knowledge, different body" is built into the architecture.

What failure looks like. A poorly fine-tuned LLM produces wrong or off-distribution text; the cost is usually mediated by a human reader. A poorly adapted VLA produces physical motion, and errors compound through a closed control loop before contacting the real world, sometimes literally. This asymmetry alone justifies treating VLA specialization as a systems and safety problem rather than a model-optimization problem.

How success is measured. Language model quality can be evaluated offline against held-out text. A robot policy's open-loop prediction error correlates only loosely with what actually matters: closed-loop task success on hardware, over long horizons, under distribution shift. Evaluation for VLAs is inseparable from simulation infrastructure and physical trials, which means the "fine-tuning loop" familiar from LLMs is really an engineering pipeline with a training step inside it.

What the data costs. Text for fine-tuning is abundant and cheap. Robot demonstrations require hardware, teleoperators, and time, which is why the entire economic logic of robot foundation models rests on making the specialization step as small as possible, and why the human-video scaling result is significant.

We should be honest about the counterargument. In NVIDIA's own tooling, adapting GR00T is called fine-tuning, and mechanically it often is: gradients flow, weights update, sometimes through much of the network. For a robot far from the pretraining distribution (an unusual morphology, a contact-rich task the model has never seen), substantial weight updates may be genuinely necessary, and the "knowledge transfer" framing gets blurrier. The distinction we're drawing is about the typical case and the correct mental model, not a claim that foundation weights are sacrosanct. But the typical industrial case really does look like adaptation: the manipulation knowledge transfers, and what's learned is the bridge.

Why this matters industrially: the arithmetic of variability

The stakes of this framing become concrete the moment you consider deployment at scale.

Picture an enterprise operating three fulfillment centers. Each has different camera models, different lighting, different shelving geometry, different robot arms from different vendors, different end effectors. Under an LLM-style mental model ("each deployment gets its own fine-tuned model"), this enterprise maintains three heavyweight training pipelines, three demonstration-collection efforts, and three divergent model lineages that drift apart with every update. Every hardware change anywhere restarts a substantial training effort. The maintenance burden grows multiplicatively with variability.

Under an adaptation mental model, the picture inverts. One foundation model, treated as a stable and infrequently-updated asset, sits at the center. Around it, lightweight adaptation layers absorb the variability: embodiment adapters that map the model's action representation onto each robot's kinematics and controller; perception adapters that normalize each facility's cameras and lighting; task adapters that specialize behavior for bin picking here and depalletizing there. When a gripper is swapped, you retrain a gripper-facing adapter. When a camera is repositioned, you touch the perception path. The foundation model, and everything it knows about physics and manipulation, stays put.

Readers familiar with parameter-efficient methods in NLP will recognize the shape of this: it resembles the adapter and LoRA ecosystem far more than it resembles classic full-model fine-tuning. But in robotics the modularity is not merely a compute optimization. It is what makes the maintenance economics tractable at all, because industrial environments change constantly (robots are upgraded, fixtures are moved, product lines turn over quarterly), and a model that must relearn manipulation every time a workcell changes is a model that will never be economically deployed.

The pattern generalizes across sectors. In automotive assembly, adapting to torque tools and precision insertion should not require touching the model's general understanding of contact and alignment. In manufacturing, where production lines reconfigure every few months, incremental demonstrations should update fixtures and end-effector adapters, not restart pretraining-scale efforts. In retail fulfillment, introducing a new picking arm changes exactly one thing, the embodiment, and the specialization cost should be proportional to that one change. NVIDIA's release of GR00T-H, a surgical variant post-trained from the same N1.7 foundation on operating-room data, is an early demonstration of the same logic at the level of an entire vertical: the physical intelligence is shared; the domain grounding is layered on top.

Vocabulary shapes architecture

It is tempting to dismiss all this as a terminology dispute. We don't think it is, because the words practitioners use shape the systems they build.

Teams that think in terms of fine-tuning tend to build monolithic pipelines: collect data, update the model, redeploy, repeat. Teams that think in terms of adaptation tend to build modular ones: a frozen or slowly-evolving foundation, a schema for normalizing data across robots and facilities, a library of adapters with independent lifecycles, and an evaluation gate, usually simulation-first, between any adapter change and the factory floor. The second architecture is more robust to exactly the kind of change industrial environments generate, and it is much easier to reason about when something goes wrong, because faults localize to the layer that changed.

Our expectation is that the field's vocabulary will catch up with this reality, much as NLP's vocabulary evolved from "fine-tuning" toward the finer-grained language of parameter-efficient adaptation, instruction tuning, and post-training. There remain genuine open questions here: when adapter-only specialization is insufficient, how to prevent adapters from silently eroding the foundation's generality, how to certify safety when the deployed policy is a composition of independently-updated modules. We don't want to understate them. But the direction seems clear. The models are learning physics once. Everything after that is a question of how efficiently we can teach an existing intelligence to inhabit a new body, in a new place, doing a new job.

That is adaptation. Calling it fine-tuning undersells what the foundation model already knows, and misleads us about where the real engineering work now lies.

Building an adaptation pipeline for GR00T N1.7 in industrial robotics

The most persistent misconception about robot foundation models is that putting one to work requires retraining a billion-parameter network. In practice, the opposite discipline produces better systems: treat the foundation model as a stable asset, and concentrate engineering effort on the interfaces: between perception and the model, between the model and the robot, between the robot and the operation it serves. This post walks through what that pipeline looks like in practice, from problem definition to production, and explains the reasoning behind each stage. We use NVIDIA's GR00T N1.7 as the running example, but the structure applies to vision-language-action models generally.

The shape of the problem

Before describing the pipeline, it's worth stating the premise it rests on, because every downstream decision follows from it.

GR00T N1.7 arrives already knowing a great deal about manipulation. Its pretraining (more than 20,000 hours of egocentric human video, large multi-robot demonstration corpora, and simulation at scale) has taught it how objects behave, how grasps succeed, and how tasks decompose. What it does not know is anything about your deployment: your robot's kinematics, your gripper, your cameras, your lighting, your parts, your workflows. The adaptation problem is therefore not "make the model smarter"; it is "connect an already-capable model to a specific body in a specific place doing a specific job, and keep that connection healthy as all three change over time."

That last clause is where most of the difficulty lives. Industrial environments are not static. Grippers get replaced, cameras get bumped, SKUs turn over, lines get reconfigured. A pipeline designed as a one-time training event will decay; a pipeline designed as a continuous adaptation loop will improve. The ten stages below are best read not as a checklist but as the anatomy of that loop.

Stage 1Define the manipulation problem precisely

Adaptation efforts fail most often at this stage, before any model is touched, because the objective was framed at the wrong granularity. "Warehouse automation" is not a task a policy can learn or an evaluation can measure. "Pick single items from mixed-SKU bins and place them into totes, at a target cycle time, with a target success rate, recovering autonomously from failed grasps" is.

The discipline is to decompose operations into atomic manipulation tasks (bin picking, tote unloading, conveyor loading, shelf replenishment, palletization, cable insertion), each with measurable success criteria defined before data collection begins. This matters for a reason beyond project management: the success criteria become the evaluation suite, and in robotics the evaluation suite is the only trustworthy signal you have. A VLA's training loss and open-loop prediction error correlate only weakly with closed-loop task success on hardware. If you cannot measure the task, you cannot know whether adaptation worked.

Stage 2Specify the embodiment

The target robot's specification (degrees of freedom, joint limits, end effector, force capabilities, camera configuration, control frequency) is not deployment paperwork. It defines the action space the adaptation must learn, and in GR00T's architecture it is a first-class concept: robots are indexed by embodiment tags, proprioceptive state is encoded through embodiment-specific projections, and the model's output vectors are shaped to the robot's degrees of freedom.

Two decisions at this stage deserve more attention than they usually get. The first is action representation: whether the policy commands joint positions or Cartesian end-effector motion, and whether targets are absolute or relative. This choice interacts with everything downstream (demonstration recording, controller design, safety limits), and changing it later means recollecting data. The second is control frequency and latency budget. GR00T's diffusion head denoises action chunks conditioned on live robot state; the horizon over which those chunks are executed, and the round-trip latency of inference, determine how reactive the deployed system can be. These are systems constraints, and they should be fixed before demonstrations are recorded so the data reflects the regime the policy will actually run in.

Stage 3Collect demonstrations deliberately

Each demonstration trajectory should synchronize everything the model will condition on at inference time: the language instruction, RGB (and optionally depth) streams from every camera, joint states, end-effector pose, gripper state, force-torque signals where available, and the executed action commands.

Volume matters less than composition. A dataset of flawless expert runs teaches a policy what success looks like but nothing about the failure modes it will inevitably encounter. Deliberately include recovery behaviors (regrasp after a slip, retreat and retry after a missed insertion) and controlled failures, across varied lighting, object placements, and clutter. In our experience the recovery demonstrations are disproportionately valuable: a policy that has never seen a failed grasp handled gracefully will, upon its first failed grasp in production, do something you did not anticipate. Because the foundation model already carries broad manipulation priors, the demonstration budget here is measured in hours or hundreds of episodes, not the thousands that training from scratch would demand. That is the entire economic point of starting from a foundation.

Stage 4Normalize into a unified schema

Industrial fleets produce chaotic data: different robots report state in different formats, cameras have different intrinsics, timestamps drift, coordinate conventions disagree, and language annotations range from careful to absent. Before any training, standardize coordinate frames, timestamp alignment, camera calibration metadata, action representations, robot state formats, and instruction phrasing into a single schema.

This stage looks like drudgery and is actually strategy. A unified schema is what makes data portable across robots, across facilities, across time. It is what allows demonstrations collected on the arm in facility A to inform the adapter for the arm in facility B, and what allows next year's replacement gripper to inherit rather than restart this year's dataset. The ecosystem is converging on standardized episode formats for exactly this reason, and GR00T's tooling expects data organized around explicit modality configurations per embodiment. Treat the schema as a durable asset with an owner, not a preprocessing script.

Stage 5Learn the embodiment adapter

With clean data in hand, the central training step is deliberately narrow: rather than updating the entire network, train the components that map the foundation model's action representation onto the target robot. In GR00T N1.7's dual-system architecture, the vision-language backbone (Cosmos-Reason2-2B) carries the general visual and semantic understanding, while the diffusion transformer head translates that understanding into motor commands conditioned on robot state. Specialization effort concentrates naturally at the embodiment-facing end: the state encodings, the action heads, the layers that bridge latent intent to this robot's joints.

What the adapter is actually learning is the residual between the pretraining distribution and your robot: kinematic differences, joint constraints, workspace limits, and, easy to underestimate, the response characteristics of your low-level controller, since the same commanded trajectory produces different motion on different control stacks. Everything general stays put.

A practical caveat: adapter-scale training is the right default, not a universal law. If your embodiment or task sits far from the pretraining distribution (an unusual morphology, contact dynamics the model has plausibly never seen), light-touch adaptation may plateau, and unfreezing more of the network becomes justified. The discipline is to earn that escalation with evaluation evidence rather than assume it, because every parameter you unfreeze is generality you may quietly erode.

Stage 6Adapt to the environment

Embodiment is only one axis of variation; the facility is the other. Lighting spectra, shelf geometry, bin dimensions, camera mounting, floor markings, and the visual character of your specific parts all shift the observation distribution away from pretraining.

The most cost-effective tools here are synthetic. Domain randomization in simulation, varying lighting, textures, camera jitter, and object appearance around a digital model of your workcell, buys robustness that would take enormous quantities of real-world data to match. Real demonstrations then anchor the policy to the true environment while the synthetic diversity prevents overfitting to any one snapshot of it. The goal is a perception pathway that treats your facility's appearance as one point in a distribution the policy already covers, so that the inevitable drift (a relamped ceiling, a repositioned camera) degrades performance gracefully rather than catastrophically.

Stage 7Specialize for the task

With the embodiment and environment handled, the final adaptation layer encodes operational workflow. A warehouse picking task decomposes into locating the SKU, estimating its pose, selecting a stable grasp that avoids disturbing neighbors, and placing into the tote; a manufacturing insertion task into locating the part, aligning, inserting, verifying contact, and releasing. N1.7's reasoning-oriented backbone is designed for exactly this kind of multi-step decomposition (the model reasons over subtasks rather than treating an instruction as an undifferentiated blob), which is what makes long-horizon workflows tractable at all.

The important property is that task adapters are thin. They sequence and specialize capabilities the foundation already has; they do not reteach manipulation. This is what makes it economical to support many workflows (picking here, kitting there, palletizing on the next line) on one shared foundation, and to add a new workflow without perturbing the ones already in production.

Stage 8Validate in simulation before hardware

No adapter change should reach a physical robot without passing a simulation gate. The gate should test at minimum: collision behavior across the workspace, proximity to kinematic singularities, end-to-end latency under realistic inference load, recovery behavior when grasps fail or objects shift, adherence to safety envelopes, and performance under stress: edge-case object poses, degraded lighting, sensor dropout.

Simulation-first validation is standard practice in robotics for a reason that bears repeating in the foundation-model era: a policy is now a composition of a large pretrained network and several independently-updated adapters, and the failure modes of a composition are harder to predict than the failure modes of any component. Sim-to-real gaps are real, and simulation success does not guarantee hardware success, but simulation failure reliably predicts hardware failure, and catching it there costs nothing but compute. Staged hardware rollout (reduced speed, supervised operation, progressively wider conditions) should follow, not replace, the simulated gate.

Stage 9Close the loop with incremental adaptation

Deployment is the beginning of the data pipeline, not the end of it. Production generates exactly the data the initial collection could not: real failed grasps, operator interventions and corrections, genuinely novel SKUs, and slow environmental drift. Route this data, curated rather than raw, back into the pipeline, and use it to refine only the adapters implicated by each change. New SKUs update task-level components; a swapped gripper updates the embodiment adapter; a lighting retrofit touches the perception path.

This surgical scope is the payoff of the modular architecture. Full retraining on every change would make continuous improvement economically impossible; adapter-scoped updates make it routine. Two disciplines keep the loop healthy. First, regression evaluation: every adapter update reruns the full task evaluation suite, because an update that fixes the new failure while breaking an old success is a net loss that offline metrics will not catch. Second, curation over accumulation: production data is heavily biased toward the common case, and naively training on all of it will erode performance on the rare cases that matter most.

Stage 10Assemble the production stack

A mature deployment arranges these pieces into a clean hierarchy: the operator and WMS/ERP systems on top, a task planner beneath them, the GR00T N1.7 foundation model at the center, the four adapters (perception, embodiment, environment, task) wrapped around it, and the low-level controller and the robot itself at the bottom.

Each layer has a distinct rate of change, and the architecture works because those rates are respected. The foundation model updates rarely, on the vendor's release cadence, with full revalidation. Adapters update on the cadence of your operation: weekly, perhaps daily. The task planner changes with business logic. The controller changes almost never. Faults localize to the layer that last changed, which is most of what makes a composed learned system debuggable. And the foundation layer remains a shared asset: the same frozen weights serve every robot, facility, and workflow, with variability absorbed entirely by the thin layers around it.

Closing perspective

The long-term trajectory of foundation robotics will be determined less by how often massive models are retrained and more by how well the industry builds adaptation pipelines around them. Industrial reality is perpetual change (robots upgraded, grippers replaced, cameras repositioned, workflows redesigned), and a foundation model should not need to relearn physics because a workcell was rearranged.

Our view is that the durable competitive advantage in this space will belong to the teams that master the pipeline, not the teams with the largest fine-tuning budgets: clean data schemas, modular adapters with independent lifecycles, simulation gates that catch composition failures before hardware does, and closed loops that turn production experience into targeted improvement. The foundation models will increasingly be shared infrastructure. The adaptation machinery around them is where the engineering, and the differentiation, now lives.