Jaehwan Jeong^1,2,*, Tuan-Anh Vu^2,*, Radha Lahoti², Jiawen Wang²,
Vivek Alumooti², Sangpil Kim^1,†, and M. Khalid Jawed<sup>2,†

1</sup>Korea University,   ²University of California, Los Angeles  

This repository provides three separate environments and codebases depending on the target usage:

1. 3D Skeletonization & Grasping Point Estimation
2. Sawyer Robot Control
3. Simulation

Follow the instructions below for each case.

This section provides the environment and code to perform plant 3D skeletonization and extract grasping points for robotic manipulation.

We identified these points by utilizing 30 corresponding pairs of the following data, all mapped within a single coordinate system: RGB, Depth (Intel RealSense D455) 7-DoF Robotic arm data (Rethink Robotics Sawyer) More Example

• Ubuntu 22.04.5 LTS
• CUDA 12.1
• Python 3.10

# Clone with submodules
git clone --recursive https://github.com/StructuresComp/robotic-pollination.git
cd robotic-pollination/3d_skeleton_grasping

# Create and activate environment
conda env create -f environment.yml
conda activate robopol

Note
This project integrates Grounded-SAM-2 as a submodule for segmentation and object grounding.

# Install Grounded-SAM-2 in editable mode
cd third_party/Grounded-SAM-2
pip install -e .

# Download pretrained SAM 2 checkpoints
cd checkpoints
bash download_ckpts.sh
cd ..

# Download pretrained Grounding DINO checkpoints
cd gdino_checkpoints
bash download_ckpts.sh
cd ../../..

Download the required datasets from UCLA Box:

• data_calibration.zip → calibration dataset for hand–eye calibration between robot and camera

Unzip them into the project directory:

# unzip data_tomato.zip -d ./data_tomato/
unzip data_calibration.zip -d ./data_calibration/

Directory Structure

robopol/
├── third_party/                      ← External dependencies (Grounded-SAM-2)
├── data_calibration/                 ← Hand–eye calibration dataset
├── data_tomato/                      ← Sample tomato plant dataset
├── environment.yml                   ← Conda environment specification
├── run.sh                            ← Script to run 1_fuse, 2_pc_clean, 3_visualization in sequence
├── 0_hand_eye_calibration.py         ← Hand–eye calibration script
├── 1_fuse_rgbd_dino_sam.py           ← RGB-D fusion + DINO + SAM integration
├── 2_pc_clean_and_skeleton.py        ← Point cloud cleaning, 3D skeletonization, and grasping point extraction
├── 3_visualization.py                ← Turn-table visualization code
└── 4_projection.py                   ← Projecting a 3D Skeleton onto a 2D Plane

Step 0. Hand–Eye Calibration (Mandatory)
You must first perform hand–eye calibration to obtain the transformation matrix T_{Camera←Flange}:

python 0_hand_eye_calibration.py

Step 1. Choose Execution Mode
Option A. Sequential Execution (Recommended)

# Before running, edit run.sh to adjust dataset paths (e.g., "./data_tomato/plant-5b")
bash ./run.sh

This will run the following steps in order:

1. 1_fuse_rgbd_dino_sam.py
2. 2_pc_clean_and_skeleton.py

Option B. Step-by-Step Debugging

python 1_fuse_rgbd_dino_sam.py
python 2_pc_clean_and_skeleton.py
python 3_visualization.py
python 4_projection.py

After running the pipeline, results are saved under the corresponding _ply/ directory.
The main output file is:

• grasping_points.json → JSON file containing detected grasping points and related vectors

Example snippet:

[
  {
    "DIR_name": "data_tomato_new/far_pepper0",                ← Dataset dir
    "plant_id": 1,                                            ← A unique identifier for the plant
    "grasping_point": [1.1040, 0.0663, 0.3649],               ← Gripper grasping location [x, y, z]
    "ideal_approach_vector": [0.3400, -0.9253, 0.1678],       ← Optimal approach vector to the grasping_point
    "constrained_approach_vector": [0.8226, 0.5432, -0.1678], ← Modified approach vector considering constraints
    // Target point 00cm away from the grasping_point
    "point_10cm": [1.0218, 0.0120, 0.3817],
    "point_30cm": [0.8572, -0.0966, 0.4152],
    // 7-DOF pose [x, y, z, qx, qy, qz, qw] at the 00cm point
    "10cm_pose_7dof": [1.0218, 0.0120, 0.3817, 0.2198, -0.7318, -0.1855, -0.6177],
    "30cm_pose_7dof": [0.8572, -0.0966, 0.4152, 0.2198, -0.7318, -0.1855, -0.6177]
  }
]

This section contains the environment setup and code required to run the Rethink Robotics Sawyer Robot.

• Ubuntu 20.04 with compatible ROS (Noetic)
• Intera SDK and sawyer_robot packages
• Network connection between your PC and the Sawyer robot controller

# 1. Create workspace
mkdir -p ~/sawyer_ws/src
cd ~/sawyer_ws/src

# 2. Clone required repositories
git clone https://github.com/RethinkRobotics/intera_sdk.git
git clone https://github.com/RethinkRobotics/sawyer_robot.git

# 3. Install dependencies
cd ~/sawyer_ws
rosdep install --from-paths src --ignore-src -r -y

# 4. Build workspace
catkin_make

# 5. Source environment
source devel/setup.bash

# 6. (Optional) Source Intera setup script
source ~/sawyer_ws/src/intera_sdk/intera.sh

# Connect to Sawyer robot
export ROS_MASTER_URI=http://<sawyer_ip>:11311
export ROS_IP=<your_pc_ip>
source ~/sawyer_ws/devel/setup.bash

# Verify communication
rosnode list
rostopic list

Simulation provides a powerful tool to predict how grasping location and vibration parameters influence flower motion, enabling informed choices that maximize pollination effectiveness. The plant dynamics are simulated using PyDiSMech. The setup instructions for the codebase can be found in the README.md file of the repository.

The input to the simulator is the skeleton mesh text file. This file contains the spatial coordinates of all the nodes and the edge connectivity array obtained from the .obj file from the 3D reconstruction. The input.txt file is structured as below:

*Nodes
1.1280794, 0.0503341, 0.2049156
1.1200794, 0.0503341, 0.2129156
1.0720794, 0.0743341, 0.2129156
...
*Edges
1, 2
2, 6
3, 4
...

Simulation examples for the plant vibration can be found in /experiments/. An example jupyter notebook to run is available at tomato_pollination_pepper0_3.ipynb. The mesh input.txt files are located in /tests/resources/tomato_pollination. The jupyter notebook has a helper function to restructure the data from the .obj into the required .txt format for the simulations.

The simulation will save the timeseries data of the DOFs (node positions) in a .js file and also create a gif of the motion of the plant for better visualization as shown below. To generate photorealistic renderings, we use Three.js.

Simulation result	After rendering

Additionally, time-series plots of the flower node position will be generated to analyze plant motion. By simulating different grasping points and vibration amplitudes and frequencies, and examining the resulting motion characteristics, we can identify the most effective vibration strategy to maximize flower motion.