Skip to main content

Unity Integration with ROS 2

Learning Objectives​

By the end of this chapter, you will be able to:

  • Set up Unity for robotics simulation
  • Integrate Unity with ROS 2 using ROS# or similar tools
  • Create digital twin environments in Unity
  • Implement bidirectional communication between Unity and ROS 2

Introduction to Unity for Robotics​

Unity is a powerful game engine that has been increasingly adopted for robotics simulation and digital twin applications. Its advantages include:

  • High-quality 3D rendering and physics
  • Extensive asset library
  • Cross-platform deployment
  • Active developer community
  • Real-time performance

Setting Up Unity for Robotics​

Prerequisites​

  • Unity Hub and Unity 2021.3 LTS or later
  • Visual Studio or another C# IDE
  • ROS 2 (Humble Hawksbill recommended)
  • Robot Operating System (ROS) connection tools

Installation of ROS# Unity Package​

ROS# is a Unity package that enables communication between Unity and ROS 2:

  1. Download the ROS# Unity package from the Unity Asset Store or GitHub
  2. Import it into your Unity project
  3. Configure the ROS connection settings

Basic Unity-ROS Communication​

Setting Up the ROS Bridge​

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using RosSharp.RosBridgeClient;

public class UnityRosConnector : MonoBehaviour
{
    public string rosBridgeServerUrl = "ws://192.168.1.100:9090"; // Adjust to your ROS bridge URL
    private RosSocket rosSocket;

    void Start()
    {
        // Connect to ROS bridge
        WebSocketNativeClient webSocket = new WebSocketNativeClient(rosBridgeServerUrl);
        rosSocket = new RosSocket(webSocket);

        Debug.Log("Connected to ROS Bridge at: " + rosBridgeServerUrl);
    }

    void OnDestroy()
    {
        rosSocket.Close();
    }
}

Publishing Messages to ROS​

using UnityEngine;
using RosSharp.RosBridgeClient;
using RosSharp.Messages.Geometry;

public class UnityPublisher : MonoBehaviour
{
    private RosSocket rosSocket;
    private string publisherId;

    void Start()
    {
        // Initialize ROS connection (assuming UnityRosConnector exists)
        GameObject connector = GameObject.FindWithTag("ROSConnector");
        if (connector != null)
        {
            rosSocket = connector.GetComponent<UnityRosConnector>().rosSocket;
            publisherId = rosSocket.Advertise<Pose>("unity_robot_pose");
        }
    }

    void Update()
    {
        // Publish robot pose every second
        if (Time.time % 1.0f < Time.deltaTime)
        {
            PublishRobotPose();
        }
    }

    void PublishRobotPose()
    {
        Pose pose = new Pose();
        pose.position.x = transform.position.x;
        pose.position.y = transform.position.y;
        pose.position.z = transform.position.z;

        // Convert Unity rotation to ROS quaternion
        pose.orientation.x = transform.rotation.x;
        pose.orientation.y = transform.rotation.y;
        pose.orientation.z = transform.rotation.z;
        pose.orientation.w = transform.rotation.w;

        rosSocket.Publish(publisherId, pose);
    }
}

Subscribing to ROS Messages​

using UnityEngine;
using RosSharp.RosBridgeClient;
using RosSharp.Messages.Geometry;

public class UnitySubscriber : MonoBehaviour
{
    private RosSocket rosSocket;
    private string subscriberId;

    void Start()
    {
        GameObject connector = GameObject.FindWithTag("ROSConnector");
        if (connector != null)
        {
            rosSocket = connector.GetComponent<UnityRosConnector>().rosSocket;
            subscriberId = rosSocket.Subscribe<Twist>("cmd_vel", ReceiveTwistCommand);
        }
    }

    void ReceiveTwistCommand(Twist twist)
    {
        // Apply the received velocity command to the Unity object
        float linearX = (float)twist.linear.x;
        float angularZ = (float)twist.angular.z;

        // Convert ROS velocity to Unity movement
        transform.Translate(Vector3.forward * linearX * Time.deltaTime);
        transform.Rotate(Vector3.up, angularZ * Time.deltaTime * Mathf.Rad2Deg);
    }
}

Advanced Unity-ROS Communication with Custom Messages​

using UnityEngine;
using RosSharp.RosBridgeClient;
using RosSharp.Messages.Std;
using RosSharp.Messages.Geometry;

public class AdvancedUnityRosInterface : MonoBehaviour
{
    [Header("ROS Connection")]
    public string rosBridgeUrl = "ws://localhost:9090";

    [Header("Robot Configuration")]
    public float maxLinearVelocity = 1.0f;
    public float maxAngularVelocity = 1.0f;

    private RosSocket rosSocket;
    private string cmdVelPublisherId;
    private string poseSubscriberId;
    private string statusPublisherId;

    void Start()
    {
        // Initialize ROS connection
        ConnectToRosBridge();

        // Set up publishers and subscribers
        SetupRosCommunication();
    }

    void ConnectToRosBridge()
    {
        try
        {
            WebSocketNativeClient webSocket = new WebSocketNativeClient(rosBridgeUrl);
            rosSocket = new RosSocket(webSocket);
            Debug.Log("Connected to ROS Bridge");
        }
        catch (System.Exception e)
        {
            Debug.LogError($"Failed to connect to ROS Bridge: {e.Message}");
        }
    }

    void SetupRosCommunication()
    {
        if (rosSocket != null)
        {
            // Publisher for velocity commands
            cmdVelPublisherId = rosSocket.Advertise<Twist>("unity_robot/cmd_vel");

            // Subscriber for pose updates
            poseSubscriberId = rosSocket.Subscribe<PoseStamped>(
                "unity_robot/pose",
                ReceivePoseUpdate
            );

            // Publisher for robot status
            statusPublisherId = rosSocket.Advertise<RosSharp.Messages.Std.String>("unity_robot/status");
        }
    }

    void Update()
    {
        // Example: Send robot status periodically
        if (Time.time % 2.0f < Time.deltaTime) // Every 2 seconds
        {
            SendRobotStatus();
        }
    }

    void SendVelocityCommand(float linear, float angular)
    {
        if (rosSocket != null)
        {
            Twist twist = new Twist();
            twist.linear = new RosSharp.Messages.Geometry.Vector3();
            twist.angular = new RosSharp.Messages.Geometry.Vector3();

            twist.linear.x = Mathf.Clamp(linear, -maxLinearVelocity, maxLinearVelocity);
            twist.angular.z = Mathf.Clamp(angular, -maxAngularVelocity, maxAngularVelocity);

            rosSocket.Publish(cmdVelPublisherId, twist);
        }
    }

    void ReceivePoseUpdate(PoseStamped poseStamped)
    {
        // Update Unity object position based on received pose
        transform.position = new Vector3(
            (float)poseStamped.pose.position.x,
            (float)poseStamped.pose.position.y,
            (float)poseStamped.pose.position.z
        );

        transform.rotation = new Quaternion(
            (float)poseStamped.pose.orientation.x,
            (float)poseStamped.pose.orientation.y,
            (float)poseStamped.pose.orientation.z,
            (float)poseStamped.pose.orientation.w
        );
    }

    void SendRobotStatus()
    {
        if (rosSocket != null)
        {
            var statusMsg = new RosSharp.Messages.Std.String();
            statusMsg.data = $"Robot at position: {transform.position}, time: {Time.time}";
            rosSocket.Publish(statusPublisherId, statusMsg);
        }
    }

    void OnDestroy()
    {
        if (rosSocket != null)
        {
            rosSocket.Close();
        }
    }

    // Public method to control the robot from other Unity scripts
    public void MoveRobot(float linearVelocity, float angularVelocity)
    {
        SendVelocityCommand(linearVelocity, angularVelocity);
    }
}

Unity-ROS Sensor Simulation​

using UnityEngine;
using RosSharp.RosBridgeClient;
using RosSharp.Messages.Sensor;

public class UnityLaserScanner : MonoBehaviour
{
    [Header("Laser Scanner Configuration")]
    public int numberOfRays = 360;
    public float scanRange = 10.0f;
    public float scanAngle = 360.0f;
    public string laserTopic = "/unity_laser_scan";
    public LayerMask detectionMask = -1; // All layers

    private RosSocket rosSocket;
    private string laserPublisherId;
    private LaserScan laserScanMsg;

    void Start()
    {
        InitializeLaserScanner();
    }

    void InitializeLaserScanner()
    {
        // Find ROS connector (assuming it exists in the scene)
        GameObject connector = GameObject.FindGameObjectWithTag("ROSConnector");
        if (connector != null)
        {
            var rosConnector = connector.GetComponent<UnityRosConnector>();
            if (rosConnector != null)
            {
                rosSocket = rosConnector.rosSocket;
                laserPublisherId = rosSocket.Advertise<LaserScan>(laserTopic);

                // Initialize laser scan message
                laserScanMsg = new LaserScan();
                laserScanMsg.angle_min = -scanAngle * Mathf.Deg2Rad / 2;
                laserScanMsg.angle_max = scanAngle * Mathf.Deg2Rad / 2;
                laserScanMsg.angle_increment = (scanAngle * Mathf.Deg2Rad) / numberOfRays;
                laserScanMsg.time_increment = 0;
                laserScanMsg.scan_time = 0.1f; // 10Hz
                laserScanMsg.range_min = 0.1f;
                laserScanMsg.range_max = scanRange;

                // Initialize ranges array
                laserScanMsg.ranges = new float[numberOfRays];
                for (int i = 0; i < numberOfRays; i++)
                {
                    laserScanMsg.ranges[i] = scanRange; // Initialize with max range
                }
            }
        }
    }

    void Update()
    {
        if (Time.time % 0.1f < Time.deltaTime) // 10Hz update
        {
            PerformLaserScan();
            PublishLaserScan();
        }
    }

    void PerformLaserScan()
    {
        float angleStep = scanAngle / numberOfRays;
        float currentAngle = -scanAngle / 2;

        for (int i = 0; i < numberOfRays; i++)
        {
            // Calculate ray direction based on current angle
            float radAngle = (currentAngle + transform.eulerAngles.y) * Mathf.Deg2Rad;
            Vector3 rayDirection = new Vector3(Mathf.Cos(radAngle), 0, Mathf.Sin(radAngle));

            // Perform raycast
            RaycastHit hit;
            if (Physics.Raycast(transform.position, rayDirection, out hit, scanRange, detectionMask))
            {
                laserScanMsg.ranges[i] = hit.distance;
            }
            else
            {
                laserScanMsg.ranges[i] = float.PositiveInfinity; // Or scanRange for max range
            }

            currentAngle += angleStep;
        }
    }

    void PublishLaserScan()
    {
        if (rosSocket != null)
        {
            laserScanMsg.header = new RosSharp.Messages.Std.Header();
            laserScanMsg.header.stamp = new TimeStamp(0, (int)(Time.time * 1e9));
            laserScanMsg.header.frame_id = "laser_frame";

            rosSocket.Publish(laserPublisherId, laserScanMsg);
        }
    }
}

Unity-ROS Joint Controller​

using UnityEngine;
using RosSharp.RosBridgeClient;
using RosSharp.Messages.Sensor;

public class UnityJointController : MonoBehaviour
{
    [System.Serializable]
    public class JointInfo
    {
        public string jointName;
        public Transform jointTransform;
        public float minAngle = -180f;
        public float maxAngle = 180f;
        public float currentAngle = 0f;
    }

    [Header("Joint Configuration")]
    public JointInfo[] joints;

    [Header("ROS Configuration")]
    public string jointStateTopic = "/unity_joint_states";

    private RosSocket rosSocket;
    private string jointStatePublisherId;
    private RosSharp.Messages.Sensor.JointState jointStateMsg;

    void Start()
    {
        InitializeJointController();
    }

    void InitializeJointController()
    {
        // Find ROS connector
        GameObject connector = GameObject.FindGameObjectWithTag("ROSConnector");
        if (connector != null)
        {
            var rosConnector = connector.GetComponent<UnityRosConnector>();
            if (rosConnector != null)
            {
                rosSocket = rosConnector.rosSocket;
                jointStatePublisherId = rosSocket.Advertise<JointState>(jointStateTopic);

                // Initialize joint state message
                jointStateMsg = new JointState();
                jointStateMsg.name = new string[joints.Length];
                jointStateMsg.position = new double[joints.Length];
                jointStateMsg.velocity = new double[joints.Length];
                jointStateMsg.effort = new double[joints.Length];

                for (int i = 0; i < joints.Length; i++)
                {
                    jointStateMsg.name[i] = joints[i].jointName;
                    jointStateMsg.position[i] = Mathf.Deg2Rad * joints[i].currentAngle;
                    jointStateMsg.velocity[i] = 0.0;
                    jointStateMsg.effort[i] = 0.0;
                }
            }
        }
    }

    void Update()
    {
        // Update joint positions from transforms and publish
        if (Time.time % 0.05f < Time.deltaTime) // 20Hz update
        {
            UpdateJointStates();
            PublishJointStates();
        }
    }

    void UpdateJointStates()
    {
        for (int i = 0; i < joints.Length; i++)
        {
            // Update joint angle based on transform rotation
            // This assumes the joint rotates around the Z-axis
            float angle = joints[i].jointTransform.localEulerAngles.z;

            // Normalize angle to -180 to 180 range
            if (angle > 180)
                angle -= 360;

            joints[i].currentAngle = Mathf.Clamp(angle, joints[i].minAngle, joints[i].maxAngle);

            // Update ROS message
            jointStateMsg.position[i] = Mathf.Deg2Rad * joints[i].currentAngle;
            jointStateMsg.header.stamp = new TimeStamp(0, (int)(Time.time * 1e9));
        }
    }

    void PublishJointStates()
    {
        if (rosSocket != null)
        {
            jointStateMsg.header.frame_id = "base_link";
            rosSocket.Publish(jointStatePublisherId, jointStateMsg);
        }
    }

    // Method to set joint angles from ROS commands (would be called from subscriber)
    public void SetJointAngles(double[] angles)
    {
        for (int i = 0; i < Mathf.Min(joints.Length, angles.Length); i++)
        {
            float angleDegrees = (float)(angles[i] * Mathf.Rad2Deg);
            joints[i].currentAngle = Mathf.Clamp(angleDegrees, joints[i].minAngle, joints[i].maxAngle);

            // Apply rotation to the joint transform
            joints[i].jointTransform.localRotation = Quaternion.Euler(
                0, 0, joints[i].currentAngle
            );
        }
    }
}

Creating Digital Twin Environments​

Environment Modeling​

Unity excels at creating realistic digital twin environments:

  1. Terrain Generation: Use Unity's terrain tools to create realistic outdoor environments
  2. Asset Integration: Import 3D models of real-world objects and facilities
  3. Lighting: Configure lighting to match real-world conditions
  4. Physics: Set up physics properties that match real-world behavior

Example: Warehouse Digital Twin​

using UnityEngine;

public class WarehouseDigitalTwin : MonoBehaviour
{
    [Header("Warehouse Configuration")]
    public int rows = 5;
    public int columns = 10;
    public float shelfWidth = 2.0f;
    public float shelfHeight = 2.0f;
    public float shelfDepth = 1.0f;

    [Header("Robot Configuration")]
    public GameObject robotPrefab;
    public Transform robotSpawnPoint;

    void Start()
    {
        CreateWarehouseEnvironment();
        SpawnRobots();
    }

    void CreateWarehouseEnvironment()
    {
        // Create shelves
        for (int r = 0; r < rows; r++)
        {
            for (int c = 0; c < columns; c++)
            {
                Vector3 position = new Vector3(
                    c * (shelfWidth + 0.5f),
                    shelfHeight / 2,
                    r * (shelfDepth + 2.0f)
                );

                GameObject shelf = GameObject.CreatePrimitive(PrimitiveType.Cube);
                shelf.name = $"Shelf_{r}_{c}";
                shelf.transform.position = position;
                shelf.transform.localScale = new Vector3(shelfWidth, shelfHeight, shelfDepth);

                // Add shelf material
                Renderer renderer = shelf.GetComponent<Renderer>();
                renderer.material.color = Color.gray;
            }
        }

        // Create floor
        GameObject floor = GameObject.CreatePrimitive(PrimitiveType.Plane);
        floor.name = "WarehouseFloor";
        floor.transform.position = new Vector3(
            (columns * (shelfWidth + 0.5f)) / 2,
            0,
            (rows * (shelfDepth + 2.0f)) / 2
        );
        floor.transform.localScale = new Vector3(
            columns * (shelfWidth + 0.5f) / 10,
            1,
            rows * (shelfDepth + 2.0f) / 10
        );
    }

    void SpawnRobots()
    {
        if (robotPrefab != null && robotSpawnPoint != null)
        {
            GameObject robot = Instantiate(robotPrefab, robotSpawnPoint.position, robotSpawnPoint.rotation);
            robot.name = "WarehouseRobot";
        }
    }
}

Advanced Unity-ROS Integration​

Custom Message Types​

To handle custom ROS message types in Unity:

using RosSharp.RosBridgeClient;
using System;

// Define custom message structure
[Serializable]
public class CustomRobotStatus : Message
{
    public const string RosMessageName = "my_robot_msgs/RobotStatus";
    public string robot_name;
    public float battery_level;
    public bool is_charging;
    public string[] active_tasks;
    public float[] joint_positions;

    public CustomRobotStatus()
    {
        this.robot_name = "";
        this.battery_level = 0.0f;
        this.is_charging = false;
        this.active_tasks = new string[0];
        this.joint_positions = new float[0];
    }
}

Sensor Simulation​

Unity can simulate various sensors for the digital twin:

using UnityEngine;

public class UnityCameraSensor : MonoBehaviour
{
    public Camera sensorCamera;
    public string rosTopic = "/unity_camera/image_raw";
    private RenderTexture renderTexture;

    void Start()
    {
        if (sensorCamera == null)
        {
            sensorCamera = GetComponent<Camera>();
        }

        // Create render texture for camera
        renderTexture = new RenderTexture(640, 480, 24);
        sensorCamera.targetTexture = renderTexture;
    }

    void Update()
    {
        // Process camera image and send to ROS
        if (Time.time % 0.1f < Time.deltaTime) // Send at 10Hz
        {
            ProcessAndSendImage();
        }
    }

    void ProcessAndSendImage()
    {
        // Capture image from render texture
        RenderTexture.active = renderTexture;
        Texture2D image = new Texture2D(renderTexture.width, renderTexture.height);
        image.ReadPixels(new Rect(0, 0, renderTexture.width, renderTexture.height), 0, 0);
        image.Apply();

        // Convert to ROS image format and send
        // (Implementation would depend on your ROS# setup)

        // Cleanup
        Destroy(image);
    }
}

Unity Asset Integration​

Importing Robot Models​

When importing robot models into Unity:

  1. Scale: Ensure models are properly scaled (1 unit = 1 meter in ROS)
  2. Axes: Convert from ROS coordinate system (X-forward, Y-left, Z-up) to Unity (X-right, Y-up, Z-forward)
  3. Joints: Set up Unity joints to match ROS URDF joint constraints
  4. Materials: Apply appropriate materials for visual fidelity

Animation and Control​

using UnityEngine;

public class RobotJointController : MonoBehaviour
{
    public Transform[] jointTransforms;
    public float[] jointLimitsMin;
    public float[] jointLimitsMax;
    private float[] currentJointPositions;

    void Start()
    {
        currentJointPositions = new float[jointTransforms.Length];
    }

    public void SetJointPositions(float[] positions)
    {
        for (int i = 0; i < jointTransforms.Length && i < positions.Length; i++)
        {
            // Apply joint limits
            float clampedPosition = Mathf.Clamp(positions[i],
                                             jointLimitsMin[i],
                                             jointLimitsMax[i]);

            // Apply rotation to joint
            jointTransforms[i].localRotation = Quaternion.Euler(0, 0, clampedPosition * Mathf.Rad2Deg);
            currentJointPositions[i] = clampedPosition;
        }
    }

    public float[] GetJointPositions()
    {
        return currentJointPositions;
    }
}

Performance Optimization​

Level of Detail (LOD)​

For complex digital twins:

using UnityEngine;

public class DigitalTwinLOD : MonoBehaviour
{
    public GameObject[] lodLevels;
    public float[] lodDistances;
    private int currentLOD = 0;

    void Update()
    {
        float distanceToCamera = Vector3.Distance(
            Camera.main.transform.position,
            transform.position
        );

        // Switch LOD based on distance
        for (int i = 0; i < lodDistances.Length; i++)
        {
            if (distanceToCamera <= lodDistances[i])
            {
                SetLOD(i);
                break;
            }
        }
    }

    void SetLOD(int lodIndex)
    {
        if (lodIndex != currentLOD && lodIndex < lodLevels.Length)
        {
            for (int i = 0; i < lodLevels.Length; i++)
            {
                lodLevels[i].SetActive(i == lodIndex);
            }
            currentLOD = lodIndex;
        }
    }
}

Troubleshooting Common Issues​

  1. Connection Issues: Verify ROS bridge is running and network connectivity
  2. Coordinate Systems: Ensure proper conversion between ROS and Unity coordinate systems
  3. Performance: Use occlusion culling and LOD for complex scenes
  4. Synchronization: Implement proper time synchronization between Unity and ROS

Exercises​

  1. Create a simple Unity scene with a robot model and connect it to ROS
  2. Implement a camera sensor in Unity that publishes images to ROS
  3. Create a digital twin of a simple room with furniture
  4. Develop a Unity interface that visualizes ROS topics in real-time

Summary​

Unity provides a powerful platform for creating digital twins and simulation environments for robotics. By properly integrating Unity with ROS 2, you can create high-fidelity simulations that bridge the gap between virtual and real-world robotics development. The combination of Unity's rendering capabilities and ROS's robotics framework enables sophisticated digital twin applications.