3D scene editing has entered the era of AI text interaction

date

type

status

slug

summary

category

icon

password

December 4, 2023 • 2 min read

by Simon Meng, See original

Due to the almost astonishing level of scene reconstruction achieved by 3D Gaussian Splatting, recent papers have begun to focus on text-guided editing. Today, I’ll share three of the most impressive examples 🐶.

1. GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting

https://buaacyw.github.io/gaussian-editor/ (Nanyang Technological University + Tsinghua University + SenseTime)

This editor supports global transfer, addition, or deletion of objects in a Gaussian field guided by text. It even features an interactive UI! It’s open-source, but I haven’t successfully set it up on either Ubuntu or Windows 11; I keep getting the same error. I’ll need to trouble the authors to check out the issue I submitted 😂.

Moreover, it seems the method for field editing is still based on the instruct nerf2nerf approach, so when transferring the entire field in the examples, it still appears somewhat blurry with artifacts, and we can’t expect it to be particularly fast.

2. GaussianEditor: Editing 3D Gaussians Delicately with Text Instructions (Yes, it’s such a coincidence that they share the same name...)

https://gaussianeditor.github.io/ (Huawei, soon to be open-source)

This version supports text-guided editing of specific areas within the Gaussian field. It also appears to use the instruct nerf2nerf method, but it segments regions highly relevant to the text description before editing, which improves speed and stability. The trade-off seems to be that it cannot perform global transfers.

3. Control4D: Efficient 4D Portrait Editing with Text

https://control4darxiv.github.io/ (Tsinghua University, soon to be open-source)

This one is even more impressive, as it not only edits Gaussian fields but also edits 4D (dynamic) Gaussian fields. From what I gathered, it uses a 4D representation called GaussianPlanes, applying plane-based decomposition in 3D space and time to structure Gaussian Splatting better, improving efficiency and robustness in 4D editing. However, the demos focus on portrait editing, so I’m unsure if it can be applied to larger scenes.