Skip to main content

NVIDIA Isaac Sim Fundamentals

Learning Objectives​

By the end of this chapter, you will be able to:

  • Understand the NVIDIA Isaac Sim platform and its capabilities
  • Set up and configure Isaac Sim for robotics simulation
  • Create and import robot models into Isaac Sim
  • Implement physics-based simulation scenarios
  • Integrate Isaac Sim with ROS 2 for robot simulation

Introduction to NVIDIA Isaac Sim​

NVIDIA Isaac Sim is a high-fidelity simulation environment for robotics built on NVIDIA Omniverse. It provides:

  • GPU-accelerated physics simulation with PhysX
  • Photorealistic rendering for sensor simulation
  • High-fidelity sensor simulation (camera, LiDAR, IMU, etc.)
  • Integration with Isaac ROS perception pipelines
  • Nav2 navigation stack compatibility
  • Support for reinforcement learning workflows

Installing and Setting Up Isaac Sim​

Prerequisites​

Before installing Isaac Sim, ensure you have:

  • NVIDIA RTX GPU with CUDA support
  • Ubuntu 20.04 or 22.04 LTS
  • NVIDIA GPU drivers (version 510 or later)
  • Docker and nvidia-docker2
  • ROS 2 Humble Hawksbill

Installation Methods​

Isaac Sim can be run in multiple ways:

  1. Docker Container (Recommended):
# Pull the Isaac Sim Docker image
docker pull nvcr.io/nvidia/isaac-sim:4.0.0

# Run Isaac Sim with proper GPU access
docker run --gpus all -it --rm \
  --network=host \
  --env "ACCEPT_EULA=Y" \
  --env "NVIDIA_VISIBLE_DEVICES=all" \
  --env "NVIDIA_DRIVER_CAPABILITIES=all" \
  --volume $HOME/isaac-sim-cache:/isaac-sim-cache \
  --volume $HOME/isaac-sim-assets:/isaac-sim-assets \
  nvcr.io/nvidia/isaac-sim:4.0.0
  1. Omniverse Launcher:
# Download Omniverse Launcher from NVIDIA Developer site
# Install Isaac Sim extension through the launcher

Basic Configuration​

# Example: Basic Isaac Sim setup script
import omni
from pxr import Gf, UsdGeom, Sdf
import carb

# Initialize Isaac Sim
def setup_isaac_sim():
    """Initialize Isaac Sim environment"""
    # Set up physics scene
    stage = omni.usd.get_context().get_stage()

    # Create physics scene
    scene_path = Sdf.Path("/World/physicsScene")
    UsdGeom.SetDefaultPrim(stage, "physicsScene")

    # Configure gravity
    from omni.physx.scripts import physicsUtils
    physicsUtils.add_physics_scene_to_stage(stage, scene_path)

    # Set gravity (standard Earth gravity)
    gravity_path = scene_path.AppendChild("gravity")
    stage.GetPrimAtPath(scene_path).GetAttribute("physxScene:gravity").Set(
        Gf.Vec3f(0.0, 0.0, -9.81)
    )

    carb.log_info("Isaac Sim environment initialized")

# Call setup function
setup_isaac_sim()

USD and Omniverse Concepts​

Universal Scene Description (USD)​

Isaac Sim uses USD as its core data model:

# Creating a USD stage
from pxr import Usd, UsdGeom, Gf, Vt

# Create a new stage
stage = Usd.Stage.CreateNew("my_robot.usd")

# Create a prim for the robot
robot_prim = stage.DefinePrim("/World/MyRobot", "Xform")

# Add a chassis to the robot
chassis_prim = stage.DefinePrim("/World/MyRobot/Chassis", "Xform")
chassis_xform = UsdGeom.Xform.Define(stage, "/World/MyRobot/Chassis")

# Set the chassis as a cube
chassis_mesh = UsdGeom.Cube.Define(stage, "/World/MyRobot/Chassis/Mesh")
chassis_mesh.GetSizeAttr().Set(1.0)

# Save the stage
stage.GetRootLayer().Save()

Physics Properties​

# Example: Adding physics properties to a rigid body
from omni.physx.scripts import physicsUtils
from pxr import PhysxSchema

def add_rigid_body_properties(prim_path):
    """Add rigid body properties to a USD prim"""
    stage = omni.usd.get_context().get_stage()

    # Get the prim
    prim = stage.GetPrimAtPath(prim_path)

    # Add rigid body API
    rigid_body_api = PhysxSchema.PhysxRigidBodyAPI.Apply(prim)

    # Set mass
    mass_api = UsdPhysics.MassAPI.Apply(prim)
    mass_api.GetMassAttr().Set(1.0)

    # Set center of mass
    mass_api.GetCenterOfMassAttr().Set(Gf.Vec3f(0.0, 0.0, 0.0))

    # Set moments of inertia
    mass_api.GetDiagonalInertiaAttr().Set(Gf.Vec3f(0.1, 0.1, 0.1))

# Apply to a cube
add_rigid_body_properties("/World/MyRobot/Chassis/Mesh")

Creating Robot Models for Isaac Sim​

Basic Robot Model​

# Creating a differential drive robot in USD
def create_differential_drive_robot(robot_name, position):
    """Create a basic differential drive robot in USD"""
    stage = omni.usd.get_context().get_stage()

    # Create robot root
    robot_path = f"/World/{robot_name}"
    robot_prim = stage.DefinePrim(robot_path, "Xform")

    # Create chassis
    chassis_path = f"{robot_path}/Chassis"
    chassis_xform = UsdGeom.Xform.Define(stage, chassis_path)

    # Add cube as chassis
    chassis_mesh = UsdGeom.Cube.Define(stage, f"{chassis_path}/Mesh")
    chassis_mesh.GetSizeAttr().Set(0.5)

    # Add materials
    material_path = f"{robot_path}/Material"
    material = UsdShade.Material.Define(stage, material_path)

    # Set position
    chassis_xform.AddTranslateOp().Set(Gf.Vec3f(*position))

    # Create left wheel
    left_wheel_path = f"{robot_path}/LeftWheel"
    left_wheel_xform = UsdGeom.Xform.Define(stage, left_wheel_path)
    left_wheel_mesh = UsdGeom.Cylinder.Define(stage, f"{left_wheel_path}/Mesh")
    left_wheel_mesh.GetRadiusAttr().Set(0.1)
    left_wheel_mesh.GetHeightAttr().Set(0.05)
    left_wheel_xform.AddTranslateOp().Set(Gf.Vec3f(0, 0.15, -0.05))

    # Create right wheel
    right_wheel_path = f"{robot_path}/RightWheel"
    right_wheel_xform = UsdGeom.Xform.Define(stage, right_wheel_path)
    right_wheel_mesh = UsdGeom.Cylinder.Define(stage, f"{right_wheel_path}/Mesh")
    right_wheel_mesh.GetRadiusAttr().Set(0.1)
    right_wheel_mesh.GetHeightAttr().Set(0.05)
    right_wheel_xform.AddTranslateOp().Set(Gf.Vec3f(0, -0.15, -0.05))

    # Add rigid body properties to all parts
    add_rigid_body_properties(f"{chassis_path}/Mesh")
    add_rigid_body_properties(f"{left_wheel_path}/Mesh")
    add_rigid_body_properties(f"{right_wheel_path}/Mesh")

    return robot_path

# Create a robot
robot_path = create_differential_drive_robot("MyRobot", [0, 0, 0.2])

Adding Joints for Articulated Robots​

# Example: Adding a revolute joint for a wheel
def add_revolute_joint(parent_path, child_path, joint_name):
    """Add a revolute joint between two bodies"""
    stage = omni.usd.get_context().get_stage()

    # Create joint
    joint_path = f"{parent_path}/{joint_name}"
    joint = PhysxSchema.PhysxRevoluteJoint.Define(stage, joint_path)

    # Set joint properties
    joint.GetAxisAttr().Set("Z")
    joint.GetParentPathAttr().Set(parent_path)
    joint.GetChildPathAttr().Set(child_path)

    # Set limits
    joint.GetLowerLimitAttr().Set(-1000.0)
    joint.GetUpperLimitAttr().Set(1000.0)

    # Set drive (for motor control)
    joint.GetEnableAngularDriveAttr().Set(True)
    joint.GetAngularDriveTypeAttr().Set("VELOCITY")
    joint.GetAngularDriveTargetVelocityAttr().Set(0.0)
    joint.GetAngularDriveDampingAttr().Set(100.0)
    joint.GetAngularDriveStiffnessAttr().Set(1000.0)

# Add joints to robot wheels
add_revolute_joint("/World/MyRobot/Chassis/Mesh", "/World/MyRobot/LeftWheel/Mesh", "LeftWheelJoint")
add_revolute_joint("/World/MyRobot/Chassis/Mesh", "/World/MyRobot/RightWheel/Mesh", "RightWheelJoint")

Isaac Sim Extensions and API​

Creating Custom Extensions​

# Example: Custom extension for robot control
import omni.ext
import omni.kit.ui
from omni.kit.menu.utils import set_menu_enabled
import omni.usd
from pxr import Gf

class RobotControlExtension(omni.ext.IExt):
    def on_startup(self, ext_id):
        print("[robot_control] Robot Control Extension Startup")

        # Create menu entry
        self._window = omni.ui.Window("Robot Control", width=300, height=300)
        self._build_ui()

    def _build_ui(self):
        with self._window.frame:
            with omni.ui.VStack():
                label = omni.ui.Label("Robot Control Panel")

                with omni.ui.HStack():
                    omni.ui.Label("Linear Vel:")
                    self._linear_vel_field = omni.ui.FloatField()
                    self._linear_vel_field.model.set_value(0.5)

                with omni.ui.HStack():
                    omni.ui.Label("Angular Vel:")
                    self._angular_vel_field = omni.ui.FloatField()
                    self._angular_vel_field.model.set_value(0.0)

                self._control_btn = omni.ui.Button("Send Command")
                self._control_btn.set_clicked_fn(self._send_command)

    def _send_command(self):
        linear_vel = self._linear_vel_field.model.get_value_as_float()
        angular_vel = self._angular_vel_field.model.get_value_as_float()

        # Here you would send the command to your robot
        print(f"Sending command: linear={linear_vel}, angular={angular_vel}")

    def on_shutdown(self):
        print("[robot_control] Robot Control Extension Shutdown")
        self._window = None

Sensor Simulation in Isaac Sim​

# Example: Adding sensors to a robot
def add_camera_sensor(robot_path, sensor_name, position, orientation):
    """Add a camera sensor to a robot"""
    stage = omni.usd.get_context().get_stage()

    # Create sensor prim
    sensor_path = f"{robot_path}/{sensor_name}"
    sensor_xform = UsdGeom.Xform.Define(stage, sensor_path)
    sensor_xform.AddTranslateOp().Set(Gf.Vec3f(*position))
    sensor_xform.AddRotateXYZOp().Set(Gf.Vec3f(*orientation))

    # Add sensor properties
    from omni.isaac.sensor.scripts import sensorUtils
    camera_sensor = sensorUtils.createRtxLidar(
        stage,
        sensor_path,
        parent=robot_path,
        translation=position
    )

    return sensor_path

def add_lidar_sensor(robot_path, sensor_name, position):
    """Add a LiDAR sensor to a robot"""
    stage = omni.usd.get_context().get_stage()

    sensor_path = f"{robot_path}/{sensor_name}"

    # Create LiDAR sensor
    from omni.isaac.range_sensor import _range_sensor
    import omni.isaac.core.utils.prims as prim_utils

    # Create sensor prim
    lidar_prim = prim_utils.create_prim(
        prim_path=sensor_path,
        prim_type="Lidar",
        translation=position,
        parent_path=robot_path
    )

    return sensor_path

Isaac Sim-ROS Bridge​

Setting up ROS Bridge​

# Example: Configuring ROS bridge for Isaac Sim
def setup_ros_bridge():
    """Set up ROS bridge for Isaac Sim"""
    import omni
    from omni.isaac.core.utils.extensions import enable_extension

    # Enable ROS bridge extension
    enable_extension("omni.isaac.ros_bridge")

    # Configure ROS topics
    import rclpy
    rclpy.init()

    # Create ROS nodes for robot control
    from geometry_msgs.msg import Twist
    from nav_msgs.msg import Odometry
    import rclpy.node

    class IsaacSimRobotController(rclpy.node.Node):
        def __init__(self):
            super().__init__('isaac_sim_robot_controller')

            # Create subscribers for robot commands
            self.cmd_vel_sub = self.create_subscription(
                Twist,
                '/cmd_vel',
                self.cmd_vel_callback,
                10
            )

            # Create publishers for robot state
            self.odom_pub = self.create_publisher(
                Odometry,
                '/odom',
                10
            )

            # Timer for publishing state
            self.timer = self.create_timer(0.1, self.publish_odom)

            # Robot state
            self.linear_vel = 0.0
            self.angular_vel = 0.0
            self.x_pos = 0.0
            self.y_pos = 0.0
            self.theta = 0.0

        def cmd_vel_callback(self, msg):
            """Handle velocity commands"""
            self.linear_vel = msg.linear.x
            self.angular_vel = msg.angular.z

            # Update robot in Isaac Sim
            self.update_robot_velocity()

        def update_robot_velocity(self):
            """Update robot velocity in Isaac Sim"""
            # This would interface with Isaac Sim physics
            pass

        def publish_odom(self):
            """Publish odometry data"""
            odom_msg = Odometry()
            odom_msg.header.stamp = self.get_clock().now().to_msg()
            odom_msg.header.frame_id = 'odom'
            odom_msg.child_frame_id = 'base_link'

            # Set position
            odom_msg.pose.pose.position.x = self.x_pos
            odom_msg.pose.pose.position.y = self.y_pos

            # Convert theta to quaternion
            from math import sin, cos
            odom_msg.pose.pose.orientation.z = sin(self.theta / 2.0)
            odom_msg.pose.pose.orientation.w = cos(self.theta / 2.0)

            # Set velocities
            odom_msg.twist.twist.linear.x = self.linear_vel
            odom_msg.twist.twist.angular.z = self.angular_vel

            self.odom_pub.publish(odom_msg)

    # Start the controller node
    controller = IsaacSimRobotController()
    rclpy.spin(controller)

# Run ROS bridge setup
# setup_ros_bridge()  # Call this in Isaac Sim environment

Best Practices for Isaac Sim​

Performance Optimization​

  1. Use Level of Detail (LOD): Reduce geometry complexity at distance
  2. Limit Physics Steps: Balance accuracy with performance
  3. Optimize Materials: Use efficient shader networks
  4. Batch Operations: Combine similar operations when possible

Scene Organization​

# Example: Proper scene organization
def organize_scene():
    """Organize Isaac Sim scene properly"""
    stage = omni.usd.get_context().get_stage()

    # Create organized hierarchy
    world_prim = stage.DefinePrim("/World", "Xform")

    # Create separate namespaces for different elements
    robots_prim = stage.DefinePrim("/World/Robots", "Xform")
    environments_prim = stage.DefinePrim("/World/Environments", "Xform")
    sensors_prim = stage.DefinePrim("/World/Sensors", "Xform")
    lights_prim = stage.DefinePrim("/World/Lights", "Xform")

    # Add physics scene
    physics_scene = stage.DefinePrim("/World/PhysicsScene", "PhysicsScene")

    # Add terrain
    terrain_prim = stage.DefinePrim("/World/Environments/Terrain", "Xform")

Quality Assurance​

  1. Validate Physics Properties: Ensure all objects have appropriate mass and collision
  2. Check Coordinate Systems: Verify ROS and USD coordinate compatibility
  3. Test Sensor Data: Validate that sensor outputs match expectations
  4. Verify Joint Limits: Ensure all joints have appropriate constraints

Exercises​

  1. Create a simple differential drive robot in Isaac Sim with proper physics properties
  2. Add a camera and LiDAR sensor to your robot model
  3. Implement a ROS bridge to control your robot from external nodes
  4. Create a simple environment with obstacles for navigation testing

Summary​

NVIDIA Isaac Sim provides a powerful simulation environment specifically designed for robotics applications. With its GPU-accelerated physics, photorealistic rendering, and seamless ROS integration, Isaac Sim enables realistic simulation for robotics development. By properly setting up the simulation environment, creating well-structured robot models, and implementing effective ROS bridges, developers can create high-fidelity simulations that bridge the gap between virtual and real-world robotics development.

Cross-References​

This module extends the simulation concepts and provides: