• Build sample(e.g., SimpleViewer)
$ cd Samples/SimpleViewer
$ make
  • Run a sample
$ cd Bin/x64-Release
$ ./SimpleViewer
  • You should now be able to see a GUI window showing the depth stream video. Here is an image showing the GUI window


  • Build sample
$ cd Samples/ClosestPointViewer
$ make
  • Run the sample
$ cd Bin/x64-Release
$ ./ClosestPointViewer

The gif shows the closest point (marked by red and blue points).



  • If the Debian Jessie Lite is used for testing, it may require the following installation for properly start the viewer.
$ sudo apt-get install libgl1-mesa-dri

Depth Stream using C++


Libaries Used:


PCL 1.8 is not available in the Ubuntu Xenial (16.04) repositories PCL 1.7 does not have the required OpenNI2 libraries to run the Orbbec Astra with PCL 1.7 also has visualization issues due to VTK bugs, which were fixed in 1.8


This was only tested in Ubuntu 16.04 with PCL 1.8.1rc2 compiled from source and OpenNI2 provided from Structure.io

Code Example Overview

  • openni_read.cpp
    • Using the OpenNI2 library, open an depth camera stream and return the number of points
  • pcd_write.cpp
    • Using the PCL library, test writing a random point cloud to a file
  • pcl_openni_viewer.cpp
    • Using the PCL and OpenNI library, open and stream a depth device


    The pcl_openni_viewer.cpp example does not work with the Orbbec Astra as you need OpenNI2

  • pcl_openni2_viewer.cpp
    • Using the PCL and OpenNI2 library, open and stream a depth device
  • pcl_visualizer.cpp
    • Using the PCL library, test generating and viewing point clouds

Setting up OpenNI2


The way structure.io has their package setup, you cannot easily install it into your system. Therefore, you need to point to the directory you extracted OpenNI2 every time you want to use the code…

Reference the OpenNI2 setup earlier in this document.

Installing PCL


You can use a pre-built .deb for installing on Ubuntu Xenial (16.04) from the following link: https://larrylisky.com/2016/11/03/point-cloud-library-on-ubuntu-16-04-lts/


If you go with the pre-built .deb, you will need to edit the CMAKE file in /usr/share/PCL-1.8. (TODO: see below) Also, with the pre-built .deb, you do not have openni2 PCL compatibility. :(

Install Prequisites:

sudo apt-get update
sudo apt-get install git build-essential linux-libc-dev
sudo apt-get install cmake cmake-gui
sudo apt-get install libusb-1.0-0-dev libusb-dev libudev-dev
sudo apt-get install mpi-default-dev openmpi-bin openmpi-common
sudo apt-get install libflann1.8 libflann-dev
sudo apt-get install libeigen3-dev
sudo apt-get install libboost-all-dev
sudo apt-get install libvtk5.10-qt4 libvtk5.10 libvtk5-dev
sudo apt-get install libqhull* libgtest-dev
sudo apt-get install freeglut3-dev pkg-config
sudo apt-get install libxmu-dev libxi-dev
sudo apt-get install mono-complete
sudo apt-get install qt-sdk openjdk-8-jdk openjdk-8-jre

Source OpenNI2 libraries:

source ~/OpenNi/OpenNi-Linux-x64-2.3/OpenNIDevEnvironment

Download and build PCL:

mkdir ~/tmp
cd ~/tmp
git clone https://github.com/PointCloudLibrary/pcl -b pcl-1.8.1rc2
cd pcl
      mkdir build
      cd build
      -DBUILD_GPU=ON -DBUILD_apps=ON -DBUILD_examples=ON \
      make -j4

Building C++ Code

source ~/OpenNi/OpenNi-Linux-x64-2.3/OpenNIDevEnvironment
cd /path/to/your/code/src
mkdir build
cd build
cmake ..
make -j4

Running C++ Examples

source ~/OpenNi/OpenNi-Linux-x64-2.3/OpenNIDevEnvironment
./visualizer -h
./openni_viewer --help
./openni_viewer -l

Depth Stream using Python and OpenCV

You can view depth data in Python from the Orbbec Astra using the OpenNI, OpenCV, and Numpy libraries. OpenNI is used to communicate with the camera. Numpy is used to manipulate the data to be displayed in a 2D window OpenCV is used to display the manipulated depth data.

Install OpenCV-Python

sudo -H pip install opencv-python

To start, you should source the Orbbec build environment from earlier:

$ source ~/OpenNi/OpenNI-Linux-x64-2.3/OpenNIDevEnvironment

Now create a new Python file and put the following code in it:

import cv2
import numpy as np
from openni import openni2
from openni import _openni2 as c_api

# Initialize the depth device
dev = openni2.Device.open_any()

# Start the depth stream
depth_stream = dev.create_depth_stream()
depth_stream.set_video_mode(c_api.OniVideoMode(pixelFormat = c_api.OniPixelFormat.ONI_PIXEL_FORMAT_DEPTH_100_UM, resolutionX = 640, resolutionY = 480, fps = 30))

# Function to return some pixel information when the OpenCV window is clicked
refPt = []
selecting = False

def point_and_shoot(event, x, y, flags, param):
                global refPt, selecting
                if event == cv2.EVENT_LBUTTONDOWN:
                                print "Mouse Down"
                                refPt = [(x,y)]
                                selecting = True
                                print refPt
                elif event == cv2.EVENT_LBUTTONUP:
                                print "Mouse Up"
                                selecting = False
                                print refPt

# Initial OpenCV Window Functions
cv2.namedWindow("Depth Image")
cv2.setMouseCallback("Depth Image", point_and_shoot)

# Loop
while True:
                # Grab a new depth frame
                frame = depth_stream.read_frame()
                frame_data = frame.get_buffer_as_uint16()
                # Put the depth frame into a numpy array and reshape it
                img = np.frombuffer(frame_data, dtype=np.uint16)
                img.shape = (1, 480, 640)
                img = np.concatenate((img, img, img), axis=0)
                img = np.swapaxes(img, 0, 2)
                img = np.swapaxes(img, 0, 1)

                if len(refPt) > 1:
                                img = img.copy()
                                cv2.rectangle(img, refPt[0], refPt[1], (0, 255, 0), 2)

                # Display the reshaped depth frame using OpenCV
                cv2.imshow("Depth Image", img)
                key = cv2.waitKey(1) & 0xFF

                # If the 'c' key is pressed, break the while loop
                if key == ord("c"):

# Close all windows and unload the depth device
  • Run the example code and click + drag on the screen to make a “selection” box. The 2D image coordinates of the selected box are printed to the terminal.

Depth Stream using Python, OpenCV, pyqtgraph, and PCL

Install Prerequisites

conda create -n py35 python=3.5 anaconda source activate py35 conda install pyqtgraph conda install pyopengl conda install pyopengl-accelerate ~/anaconda2/envs/py35/bin/pip install opencv-python ~/anaconda2/envs/py35/bin/pip install openni
pip install opencv-python pip install openni pip install pyqtgraph pip install PyOpenGL conda install python-pcl sudo -H pip3 install PyQt5 sudo apt install python3-pyqt5.qtopengl python3-pyopengl

Code Setup

Depth Visualization using Python QT


Work in progress, incomplete.

sudo -H pip2 install pyqtgraph

Depth Visualization using Python PCL


Work in progress, incomplete