Recording Kinect One Stream using C#

kinect Aug 14, 2016

During my PhD I had to obtain three of the data streams (RGB, Depth and Skeleton) of the Kinect One sensor. Sadly, Microsoft, at the time, did not provide any example for recording the raw data itself. They do offer Kinect Studio, however it’s a closed ecosystem without the ability of accessing the raw data.

In this blog post I wanted to share with you the C# code snippets for how I recorded the data streams and achieved near 30ftp while saving these streams.

You can see an example of the code over on my GitHub page here - please note that the code it is not optimised in any way. The code is very basic and just an example. I would recommend taking the below examples and encapsulating them into a function.

Depth

Image of the depth sensor stream

The Depth Map Image of a single frame is made up of an x and y array of pixels which represents the distance (in millimetres) from the camera plane to the nearest object/person at a resolution of 640x480.

To save each Depth Map Image as they arrive from the sensor you can use the following C# code snippet embedded in the Reader_FrameArrivedDepth function.

long milliseconds = DateTime.Now.Ticks /TimeSpan.TicksPerMillisecond;

string filePath = "depth" + milliseconds + ".bin";

using (FileStream streamDepth = new FileStream(filePath, FileMode.Create))
{
using (BinaryWriter depthWriter = new BinaryWriter(streamDepth))
{
depthWriter.Write(this.pixelsDepth);
depthWriter.Close();
}
}

If we decompose the code segment in to basic parts it is actually very straightforward. Firstly, we want to assign a unique file name to each image as we save it. However, we should save each Depth Map Image with the context that they are captured in, therefore we use:

“DateTime.Now.Ticks / TimeSpan.TicksPerMillisecond” 

to define the time the Depth Map Image was save and convert this to a file path called “string filePath”. This is important if we wanted to review frame rate or latency of the recording. The final stage of this code is to save the Depth Map Image using the BinaryWriter function:

using (FileStream streamDepth = new FileStream(filePath, FileMode.Create))
{
using (BinaryWriter depthWriter = new BinaryWriter(streamDepth))
{                                        
depthWriter.Write(this.pixelsDepth);                                       
depthWriter.Close();
}

In this part we create a new stream to create the file and write the binary Depth Map Image to a .bin file.

RGB

Example of the RGB Stream

Saving the RGB image is not as straightforward as the Depth Map Image. This is due to the Kinect One sensor capturing images in High Definition. This means we need to handle these images in a specific way, otherwise our frame rate is hugely impacted due to attempting to buffer and save HD images.

To save each HD image as they arrive from the sensor we can use the following snippet embedded in the Reader_FrameArrivedColour:

{
long milliseconds = DateTime.Now.Ticks / TimeSpan.TicsPerMillisecond;

Task.Factory.StartNew(() =>
{
string filePath = "rgb" + milliseconds + ".bin";

using (FileStream streamRGB = new FileStream(filePath, FileMode.Create))
{
using (BinaryWriter rgbWriter = new BinaryWriter(streamRGB))
{
rgbWriter.Write(this.pixels);
rgbWriter.Close();
}
}
}

We again use the computer clock to generate a unique timestamp file name. Then, we define a new thread task to handle the actual saving:

Task.Factory.StartNew(() =>

The idea behind defining this new thread is to ensure that we do not impact the actual rendering of the content to the user. Further, threads will be disposed of as soon as the image has been saved. The remaining code follows the same principles used for the Depth Map Image. We then use a similar BinaryWriter to save the image to a .bin file.

Skeleton

Example of the skeleton output

Obtaining and saving the Kinect skeleton(s) is a straightforward and simple task. Below is a code snippet example. In essence, all we need to do is loop through the array that is holding the skeleton(s) and joint information.

string filePath = timeStamp + ".txt";

StreamWriter cooStream = new StreamWriter(filePath, false);

IReadOnlyDictionary<JointType, Joint> joints = body.Joints;

Dictionary<JointType, Point> jointPoints = new Dictionary<JointType, Point>();

foreach (JointType jointType in joints.Keys)
{
ColorSpacePoint depthSpacePoint = this.coordinateMapper.MapCameraPointToColorSpace(joints[jointType].Position);

cooStream.WriteLine(joints[jointType].JointType + " " + joints[jointType].TrackingState + " " + joints[jointType].Position.X + " " + joints[jointType].Position.Y + " " + joints[jointType].Position.Z + " " + depthSpacePoint.X + " " + depthSpacePoint.Y);
}

string wrtLineData = "LeftHand " + body.HandLeftState + " RightHand " + body.HandRightState;

cooStream.WriteLine(wrtLineData);
}

cooStream.Close();
        

There are some important aspects to note from the above code snippet. First, we create a StreamWriter to write each frame to a text file:

string filePath = timeStamp + ".txt";

StreamWriter cooStream = new StreamWriter(filePath, false);        

We then obtain the skeleton(s) and the associated joint information for each skeleton. With this code we are able to record all the skeletons tracked by the Kinect sensor.

IReadOnlyDictionary<JointType, Joint> joints = body.Joints;

Dictionary<JointType, Point> jointPoints = new Dictionary<JointType, Point>();        

The next step is to take the dictionary of skeletons and loop through to obtain the correct skeletal information:

foreach (JointType jointType in joints.Keys)
{
ColorSpacePoint depthSpacePoint = this.coordinateMapper.MapCameraPointToColorSpace(joints[jointType].Position);

cooStream.WriteLine(joints[jointType].JointType + " " + joints[jointType].TrackingState + " " + joints[jointType].Position.X + " " + joints[jointType].Position.Y + " " + joints[jointType].Position.Z + " " + depthSpacePoint.X + " " + depthSpacePoint.Y);
}  

The above outputs, to a text file the skeleton information and the associated Depth Map Image coordinates. The final part is to record the hand state. The Kinect is able to track the state of the hands and we obtain that by the follow:

string wrtLineData = "LeftHand " + body.HandLeftState + " RightHand " + body.HandRightState;        

The above outputs the state of the hands into a text file. We then simply close the data stream.

Tags