3D Object Recognition Using Global Descriptor (Ensemble of Shape Functions) via PCL

[muneeb127]

I am developing a system for 3D object recognition using the point cloud library. I want to apply an ESF descriptor to a data set that I have stored locally. I am giving the folder containing data to the program as an argument. I am attaching the code for reference.

#include <pcl/io/pcd_io.h>
#include <pcl/features/esf.h>
#include <pcl/visualization/histogram_visualizer.h>
#include <pcl/visualization/pcl_plotter.h>
#include <pcl/point_types.h>
#include <pcl/point_cloud.h>
#include <pcl/console/print.h>
#include <boost/filesystem.hpp>
#include <boost/range/iterator_range.hpp>
#include <flann/flann.h>
#include <fstream>

typedef std::pair<std::string, std::vector<float> > esf_model;


void calculateESFDescriptor(const boost::filesystem::path &path){
    pcl::console::print_highlight ("Executing function: calculateESFDescriptor() \n");

    // const boost::filesystem::path &path = base_dir/it->path().filename();

    // Cloud for storing the object.
    pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
    // Object for storing the ESF descriptor.
    pcl::PointCloud<pcl::ESFSignature640>::Ptr descriptor(new pcl::PointCloud<pcl::ESFSignature640>);

    // Note: you should have performed preprocessing to cluster out the object
    // from the cloud, and save it to this individual file.

    // Read a PCD file from disk.
    if (pcl::io::loadPCDFile<pcl::PointXYZ>(path.string(), *object) != 0)
    {
        pcl::console::print_highlight ("Unable to read .pcd file from the directory\n");
        // return -1;
    }

    // ESF estimation object.
    pcl::ESFEstimation<pcl::PointXYZ, pcl::ESFSignature640> esf;
    esf.setInputCloud(object);

    esf.compute(*descriptor);

    // Writing esf description file to disk
    std::string output_file_name = path.string().substr(0, path.string().find_last_of('.')) + "_esf.pcd";
    pcl::io::savePCDFileASCII (output_file_name, *descriptor);
    std::cerr << "Saved data points to " << output_file_name << std::endl;
    // return (0);
}


/** \brief Load a set of ESF features that will act as the model (training data)
  * \param argc the number of arguments (pass from main ())
  * \param argv the actual command line arguments (pass from main ())
  * \param extension the file extension containing the ESF features
  */
void traverseDirectory(const boost::filesystem::path &base_dir, const std::string &extension){
    pcl::console::print_highlight ("Executing function: traverseDirectory() \n");

    if (!boost::filesystem::exists(base_dir) && !boost::filesystem::is_directory(base_dir)){
        pcl::console::print_highlight ("Quitting as the directory does not exists.\n");
        return;
    }

    int i = 0;

    for (boost::filesystem::directory_iterator it (base_dir); it != boost::filesystem::directory_iterator (); ++it)
    {
        if (boost::filesystem::is_directory(it->status()))
        {
            std::stringstream ss;
            ss << it->path ();
            pcl::console::print_highlight ("Loading %s \n", ss.str().c_str());
            traverseDirectory(it->path(), extension);
        }   


        // Calculate ESF Descriptor for each file
        if (boost::filesystem::is_regular_file(it->status()) && boost::filesystem::extension(it->path())==extension)
        {
            calculateESFDescriptor(base_dir/it->path().filename());
            pcl::console::print_highlight ("Loop execution: %d\n", i++);
        }
        // continue;
    }
}

int main(int argc, char** argv)
{   
    if (argc < 2)
    {
        PCL_ERROR ("Need at least two parameters! Syntax is: %s [model_directory] [options]\n", argv[0]);
        return (-1);
    }   

    std::string extension(".pcd");
    transform(extension.begin(), extension.end(), extension.begin(), (int(*)(int))tolower);

    // std::vector<esf_model> models;

    // Traversing directory to find all .pcd files and applying esf descriptor to each file
    traverseDirectory(argv[1], extension);
    return (0);
}

The problem is that the function calculateESFDescriptor() runs only once and then the program shuts down without any error, whereas I have multiple .pcd files in the directory and the function should execute for all the files in the directory. I want to calculate the ESF descriptor for all the files to compare them with user-given input at a later stage to calculate the accuracy. I cannot debug where I am making the mistake due to which the program cannot execute for all the files in the directory. The program runs for the whole directory when I comment out the following line. calculateESFDescriptor(base_dir/it->path().filename());

The question is if it is possible to compute descriptor for the whole directory at once or do I have to give each file as input and run the program for each file separately?

[mvieth]

@muneeb127 I would change calculateESFDescriptor so that it receives a const std::string& filename instead of a boost::filesystem::path. Then, as a test, call it multiple times from main with fixed file names (change the file names to match actual files on your computer):

int main() {
  calculateESFDescriptor("/path/to/file/one.pcd");
  calculateESFDescriptor("/path/to/file/two.pcd");
  calculateESFDescriptor("/path/to/file/three.pcd");
  calculateESFDescriptor("/path/to/file/four.pcd");
}

Does this process the given files correctly?

[muneeb127]

Hey, I am still facing the same situation when I give the file name as input as a string and directly call the function calculateESFDescriptor(). Following is the new code as per your suggestion.

#include <pcl/io/pcd_io.h>
#include <pcl/features/esf.h>
#include <pcl/visualization/histogram_visualizer.h>
#include <pcl/visualization/pcl_plotter.h>
#include <pcl/point_types.h>
#include <pcl/point_cloud.h>
#include <pcl/console/print.h>
#include <boost/filesystem.hpp>
#include <boost/range/iterator_range.hpp>
#include <flann/flann.h>
#include <fstream>
#include <string>

bool calculateESFDescriptor(const std::string& filename){

	// Cloud for storing the object.
	pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
	// Object for storing the ESF descriptor.
	pcl::PointCloud<pcl::ESFSignature640>::Ptr descriptor(new pcl::PointCloud<pcl::ESFSignature640>);

	// Read a PCD file from disk.
	if (pcl::io::loadPCDFile<pcl::PointXYZ>(filename.c_str(), *object) != 0)
	{
		pcl::console::print_highlight ("Unable to read .pcd file from the directory\n");
		// return -1;
		return(false);
	}

	try{
		pcl::ESFEstimation<pcl::PointXYZ, pcl::ESFSignature640> esf;
		esf.setInputCloud(object);
		esf.compute(*descriptor);
		// Writing esf description file to disk
		std::string output_file_name = filename.substr(0, filename.find_last_of('.')) + "_esf.pcd";
		pcl::io::savePCDFileASCII (output_file_name, *descriptor);
		std::cerr << "Saved data points to " << output_file_name << std::endl;
		return (true);
	}
	catch (const pcl::InvalidConversionException&)
        {
              pcl::console::print_highlight ("Unable to compute....");
              return (false);
        }
}

int main(int argc, char** argv)
{	
	if (argc < 2)
	{
		PCL_ERROR ("Need at least two parameters! Syntax is: %s [model_directory] [options]\n", argv[0]);
		return (-1);
	}	

	std::string extension(".pcd");
	transform(extension.begin(), extension.end(), extension.begin(), (int(*)(int))tolower);

	// std::vector<esf_model> models;

	// Traversing directory to find all .pcd files and applying esf descriptor to each file
	calculateESFDescriptor(".\\data\\000.580.67\\1258250240333_cluster_0_nxyz.pcd");
	calculateESFDescriptor(".\\data\\000.580.67\\1258250242333_cluster_0_nxyz.pcd");
	calculateESFDescriptor(".\\data\\000.580.67\\1258290249333_cluster_0_nxyz.pcd");
	calculateESFDescriptor(".\\data\\000.580.67\\1258570238333_cluster_0_nxyz.pcd");
	return (0);
}

Following is the output in the terminal.

Still the same issue persist that the function is running for the first file only and then the program shutsdown without giving any error.

[mvieth]

@muneeb127 I tested the code from your last comment, only changing

calculateESFDescriptor("/home/mvieth/pcl/test/bun01.pcd");
calculateESFDescriptor("/home/mvieth/pcl/test/bun02.pcd");
calculateESFDescriptor("/home/mvieth/pcl/test/bun03.pcd");
calculateESFDescriptor("/home/mvieth/pcl/test/bun4.pcd");

and it printed

Saved data points to /home/mvieth/pcl/test/bun01_esf.pcd
Saved data points to /home/mvieth/pcl/test/bun02_esf.pcd
Saved data points to /home/mvieth/pcl/test/bun03_esf.pcd
Saved data points to /home/mvieth/pcl/test/bun4_esf.pcd

So it works as expected on my computer.

• Which operating system do you use?
• Which PCL version do you use? How did you install it?
• Which compiler do you use?

Can you try to find out, where exactly the program exits, either with a debugger or with std::cout prints? Does it return to main after running calculateESFDescriptor for the first time? Does it perhaps even enter calculateESFDescriptor for a second time, but exit inside the function for some reason?

[muneeb127]

@mvieth Following are the specifications of my machine.

• Windows 11 Home
• PCL 1.13.1 installed using all-in-one installer following the steps from this link
• Using CMake to compile the program

The program exits after saving the file to disk. It is not returning to main and entering the function the second time. If I am removing the functionality inside the function calculateESFDescriptor() and just printing a single statement in this function, the program is running as expected. See the code below for reference.

bool calculateESFDescriptor(const std::string& filename){
    pcl::console::print_highlight ("Filename: %s \n", filename.c_str());
    return (true);
}

The above program will run four times as expected.

[mvieth]

@muneeb127 Okay, if you have a minimal function that works, then you can add back the code piece by piece until it no longer works (that is, until it no longer runs four times as expected). For example, first add the loadPCDFile, if that works, add the ESF estimation, ...
Please also post your CMakeLists.txt

[muneeb127]

@mvieth The line which is making the difference is the esf.compute(*descriptor);. If I comment out this line, the code runs as expected and gives the following output:

Following is my CMakeLists.txt

cmake_minimum_required(VERSION 3.5 FATAL_ERROR)

project(cluster_extraction)

find_package(PCL 1.2 REQUIRED)

include_directories(${PCL_INCLUDE_DIRS})
link_directories(${PCL_LIBRARY_DIRS})
add_definitions(${PCL_DEFINITIONS})

add_executable (cluster_extraction cluster_extraction.cpp)
target_link_libraries (cluster_extraction ${PCL_LIBRARIES})

add_executable (cloud_viewer cloud_viewer.cpp)
target_link_libraries (cloud_viewer ${PCL_LIBRARIES})

add_executable (esf_estimation esf_estimation.cpp)
target_link_libraries (esf_estimation ${PCL_LIBRARIES} )

add_executable (esf2 esf2.cpp)
target_link_libraries (esf2 ${PCL_LIBRARIES} )

add_executable (read_file read_file.cpp)
target_link_libraries (read_file ${PCL_LIBRARIES} )

The problem lies somewhere in the computation of the esf descriptor, which just stops the program after running for once.

[mvieth]

@muneeb127 Can you put the ESFEstimation object into an extra scope, like this:

{
  pcl::ESFEstimation<pcl::PointXYZ, pcl::ESFSignature640> esf;
  esf.setInputCloud(object);
  esf.compute(*descriptor);
}
std::cout << "ESFEstimation object destroyed" << std::endl;

I want to test whether the program exits when that object is destroyed.
Also, how large are your PCD files (how many megabytes)? How many points do they contain?

[muneeb127]

@mvieth I am using the files from the dataset vfh_recognition_tutorial_data.tbz available in the tutorial . Normally, the file in this dataset contains around 3500 points and the average size of files is around 400 KB. I have also tried for other .pcd files, but the code also does not work for them.

Moreover, I tried changing the code as per your suggestion which is as follows:

bool calculateESFDescriptor(const std::string& filename){
	pcl::console::print_highlight ("Filename: %s \n", filename.c_str());

	// Cloud for storing the object.
	pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
	// Object for storing the ESF descriptor.
	pcl::PointCloud<pcl::ESFSignature640>::Ptr descriptor(new pcl::PointCloud<pcl::ESFSignature640>);

	// Read a PCD file from disk.
	// if (pcl::io::loadPCDFile<pcl::PointXYZ>(path.string(), *object) != 0)
	if (pcl::io::loadPCDFile<pcl::PointXYZ>(filename.c_str(), *object) != 0)
	{
		pcl::console::print_highlight ("Unable to read .pcd file from the directory\n");
		// return -1;
		return(false);
	}

	// ESF estimation object.
	// pcl::ESFEstimation<pcl::PointXYZ, pcl::ESFSignature640> esf;

	try{
		pcl::ESFEstimation<pcl::PointXYZ, pcl::ESFSignature640> esf;
		esf.setInputCloud(object);
		esf.compute(*descriptor);
		// Writing esf description file to disk
		std::string output_file_name = filename.substr(0, filename.find_last_of('.')) + "_esf.pcd";
		pcl::io::savePCDFileASCII (output_file_name, *descriptor);
		std::cerr << "Saved data points to " << output_file_name << std::endl;
		// return (true);
	}
	catch (const pcl::InvalidConversionException&)
    {
		pcl::console::print_highlight ("Unable to compute....");
        return (false);
    }

	std::cerr << "ESFEstimation object destroyed" << std::endl;

	return (true);
	// return (0);
}

Following is the output:

Still, nothing after the try block gets executed.

[muneeb127]

@mvieth Is there any tutorial related to this where a particular descriptor is being calculated for multiple files or a complete dataset?

[muneeb127]

@mvieth If I am calculating GRSD instead of ESF, the program is working as expected and giving four files as output. What could be the case for this? I think the problem lies somewhere in the computation of ESF in PCL.

#include <pcl/io/pcd_io.h>
#include <pcl/features/esf.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/grsd.h>

int
main(int argc, char** argv)
{

	std::vector<std::string> pointCloudFiles = {".\\data\\000.580.67\\1258250240333_cluster_0_nxyz.pcd", 
        ".\\data\\000.580.67\\1258250242333_cluster_0_nxyz.pcd", ".\\data\\000.580.67\\1258290249333_cluster_0_nxyz.pcd", 
        ".\\data\\000.580.67\\1258570238333_cluster_0_nxyz.pcd"};
	for (const std::string& fileName : pointCloudFiles)
    {
        // Read the point cloud data
        pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
		// Object for storing the normals.
		pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
		// Object for storing the GRSD descriptors for each point.
		pcl::PointCloud<pcl::GRSDSignature21>::Ptr descriptors(new pcl::PointCloud<pcl::GRSDSignature21>());

		// Read a PCD file from disk.
		if (pcl::io::loadPCDFile<pcl::PointXYZ>(fileName, *cloud) != 0)
		{
			return -1;
		}

		// Note: you would usually perform downsampling now. It has been omitted here
		// for simplicity, but be aware that computation can take a long time.

		// Estimate the normals.
		pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
		normalEstimation.setInputCloud(cloud);
		normalEstimation.setRadiusSearch(0.03);
		pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
		normalEstimation.setSearchMethod(kdtree);
		normalEstimation.compute(*normals);

		// GRSD estimation object.
		pcl::GRSDEstimation<pcl::PointXYZ, pcl::Normal, pcl::GRSDSignature21> grsd;
		grsd.setInputCloud(cloud);
		grsd.setInputNormals(normals);
		grsd.setSearchMethod(kdtree);
		// Search radius, to look for neighbors. Note: the value given here has to be
		// larger than the radius used to estimate the normals.
		grsd.setRadiusSearch(0.05);
		
		grsd.compute(*descriptors);

        // Store the computed descriptor (for demonstration purposes, you can choose a more suitable data structure or file storage)
        std::string output_file_name = fileName.substr(0, fileName.find_last_of('.')) + "_grsd.pcd";
        pcl::io::savePCDFileASCII(output_file_name, *descriptors);
		std::cerr << "Saved data points to " << output_file_name << std::endl;
    }
	return 0;
}

Output:

[mvieth]

@muneeb127 My best guess is that one of the two members of ESFEstimation causes a crash when ESFEstimation is destroyed after the members are filled in esf.compute(...). Can you run your program with a debugger? Otherwise I can't really help you further.

[muneeb127]

I ran the code in VS 2022 for debugging it and I found this. If I add a breakpoint on the line esf.compute()

The program gives the following error in Memory.h file:

After this error, the program does not continue and on stopping the execution, following result is printed in the cmd: