The official implementation of SAGA (Segment Any 3D GAussians).

The installation of SAGA is similar to 3D Gaussian Splatting.

or

Then install the dependencies:

In default, we use the public ViT-H model for SAM. You can download the pre-trained model from here and put it under ./third_party/segment-anything/sam_ckpt.

The used datasets are 360_v2, nerf_llff_data and LERF.

The data structure of SAGA is shown as follows:

Since we need the pre-trained 3D-GS model for mask scales extraction, the first step is to train the 3D Gaussians:

We inherit all attributes from 3DGS, more information about training the Gaussians can be found in their repo.

Then, to get the sam_masks and corresponding mask scales, run the following command:

Note that sometimes the downsample is essential due to the limited GPU memory.

If you want to try the open-vocabulary segmentation, extract the CLIP features first:

Currently SAGA provides an interactive GUI (saga_gui.py) implemented with dearpygui and a jupyter-notebook (prompt_segmenting.ipynb). To run the GUI:

Temporarily, open-vocabulary segmentation is only implemented in the jupyter notebook. Please refer to prompt_segmenting.ipynb for detailed instructions.

After setting up the GUI, you can see the following interface:

After selecting the interest target(s). You can click segment3D to get the 3D segmentation results. If the results is not satisfied, you can click roll back to undo this segmentation or click clear to roll back to the unsegmented status, or you can continue to input prompts to conduct segmentation based on the temporary segmentation result. You can click save as to save the current segmentation results in ./segmentation_res/your_name.pt, which is a binary mask for all 3D Gaussians in the 3D-GS model.

At any time, you can click cluster3d to get the clustering results of the current 3D-GS model. For example, you can directly cluster across the whole scene or cluster in the temporarily segmented objects for decomposition. Click reshuffle_cluster_color to shuffle the rendering colors of the clusters.

Note that directly clustering the whole scene may take a while, since we use HDBSCAN without the GPU support for convenience.

After saving segmentation results in the interactive GUI or running the scripts in prompt_segmenting.ipynb, the bitmap of the Gaussians will be saved in ./segmentation_res/your_name.pt (you can set the name by yourself). To render the segmentation results on training views (get the segmented object by removing the background), run the following command:

To get the 2D rendered masks, run the following command:

You can also render the pre-trained 3DGS model without segmentation:

If you find this project helpful for your research, please consider citing the report and giving a ⭐.

The implementation of saga refers to GARField, OmniSeg3D, Gaussian Splatting, and we sincerely thank them for their contributions to the community.