How to Use Microsoft Kinect with Unity

 

Last year, Microsoft and Unity created a strategic partnership and there has been significant collaboration and support between these two companies. One of the tools that emerged through this partnership was the added Kinect plugin for Unity that allows game developers to add motion controls to their games or simulations with ease. This is a brief overview of those features that are included in the plugin as well as quick step-by-step guide to aid those who are still unfamiliar with the initial configuration but wish to add motion controls using the Microsoft Kinect. All of the information I’m about to provide is a summary of this Unity wiki page.

The first step is to download the following files

Note: These scripts take advantage of a plugin feature only available for Unity Pro users.

Initial Setup

Once these files are finished downloaded, ensure that the Kinect device is not plugged into your machine, begin the installation of the Windows SDK and the Windows Developer Toolkit, and follow the on-screen prompts.

After the installation, we are ready to create a new Unity project. Name it KinectDemo or something similar to easily distinguish your project from older ones. Once Unity loads the default scene you can begin importing the assets from the Unity package that you’ve downloaded earlier.  While the project is currently open, extract the file and double click on the Unity package to start the importing process.

Once the package is finished importing you can safely plug in your Microsoft Kinect. Locate and find KinectSample scene that was provided. It is recommended to play and test this scene out to see if Kinect drivers were installed correctly. If it was successful, you should notice a RGB display screen, a grayscale depth screen, a rainbow colored humanoid model, and various spheres all translating in conjunction with your movements in real time.

If you are still having problems, uninstall the Kinect for Windows SDK and Kinect for Windows Developer Toolkit, unplug your Kinect, and retry the installation process again as described above. If there are no problems found, you are now ready to begin developing using the Microsoft Kinect and Unity.

Further Details

Taking a closer look at the project hierarchy within the KinectSample scene, you might have noticed two specific game objects labeled rainbowMan_v6 and KinectPointMan. Within those prefabs are two scripts that are responsible for capturing and mimicking your movements. These scripts are the KinectPointController and KinectModelControllerV2.

Both script require a skeleton wrapper reference to work. The KinectPrefab is the gameobject that needs to be referenced in the Sw placeholder in the inspector. This KinectPrefab will record motion data in real time and send the information to the controller scripts. The controllers scripts will convert this data and transform bones or Game objects that are related to specific parts of the body and mimic the relative placement in a 3d environment.

The Main difference between the KinectPointController and the KinectModelControllerV2 is how they reference each specific part of the body. The KinectPointController uses individual game objects. In the KinectSample scene they use spheres to represent the contact points. In practice you could replace these objects with a simple hand model or a pointer object that the user could control and interact with.

The KinectModelControllerV2 uses one specific rigged character model. A rigged model contains a digital skeleton bound to a 3D mesh. Expand your custom model fully so that each bone is visible in the hierarchy. Each digital bone that needs to be controlled must be referenced into the appropriate slot in the inspector. (See Video Below) After adding and configuring the KinectModelController, if you are having inaccuracies with your custom model (odd mesh behavior, joints in unrealistic locations) double check the placement of the bones within the rigged model as well as the weight distribution. For more information about creating custom models, explore tutorial and training videos for Blender or Maya.

Additional Features

The mask will filter and limit what joints need to be animated through the Kinect. Any bones or game objects that are excluded from this list will remain static. Therefore, if you want a game only using your hands and head, you can limit the range to track this by setting the mask to “Upper_Body”.

The player field specifies which user controls the model. To quickly add local multiplayer support, it can be as simple as duplicating your player model and changing the player value inside the inspector. The KinectPointman has scale will transform the size of all game objects that are referenced and used to translate player movements.

On the KinectModelControllerV2, if you intend to animate your model while the player is controlling it, set “animated” flag and determine how much blending between the animation and the Kinect should occur – this is a range from 0 – 1.

Quick review

  1. Before we begin, ensure that the Microsoft Kinect is NOT plugged into the machine before installation.
  2. Download the required files.
  3. Run the installer for the SDK and Developer Toolkit and follow the on-screen prompts.
  4. Create a new project within Unity and make sure to change it from the default project name.
  5. Extract the Kinect Wrapper Package for Unity and double click on the file to import all assets into the newly created project.
  6. Plug in the Microsoft Kinect device into the computer’s USB port.
  7. Inside the newly imported package is a KinectSample scene, open the scene and play it to verify motion controls are enabled.
  8. Develop the next big motion control simulation or game!

– Matt NeCamp