Question about the camera_augment

[Gatsby23]

Dear Luigi Piccinelli：
Thank you for releasing the Unik3D library. It’s a significant contribution to the AR/VR field, and really helps a lot of people.
Recently, I'm researching the camera_augment.py function within the library. However, I noticed that it lacks corresponding configuration files and other necessary materials, like 'camera_sampler.json', which has made it difficult for me to understand how to properly run and test the function.

Could you kindly provide these files or offer additional guidance on how to set up and test the function?

Thank you for your help, and for your continued contributions to the community!

Yours
Qi Wu
SJTU

[lpiccinelli-eth]

Hi, thank you for your interest! I have not added it to the repo as I need to check some old commits of my dev repo, the exact camera sampler I used for the experiments in the paper.
Anyway, I attached here one example that should resemble the one in the paper.
I will get the original one and update the repo accordingly.
Let me know if everything works fine or if any issues arise.

camera_sampler.json

[Gatsby23]

Hi, thank you for your interest! I have not added it to the repo as I need to check some old commits of my dev repo, the exact camera sampler I used for the experiments in the paper. Anyway, I attached here one example that should resemble the one in the paper. I will get the original one and update the repo accordingly. Let me know if everything works fine or if any issues arise.

camera_sampler.json

OK，thank you very much, I'll try it as soon as possible.

[Gatsby23]

Hi, thank you for your interest! I have not added it to the repo as I need to check some old commits of my dev repo, the exact camera sampler I used for the experiments in the paper. Anyway, I attached here one example that should resemble the one in the paper. I will get the original one and update the repo accordingly. Let me know if everything works fine or if any issues arise.

camera_sampler.json

Thank you for your quick response. Based on your suggestions, I was able to successfully create the camera_sampler. However, I have a few follow-up questions:

1. How should I construct and organize the input parameters for the augment_camera function? More specifically, how should I handle inputs[depth_guidance]? Could you please give me a demo about the augment_camera usage?

def augment_camera(self, inputs, camera_sampler):
    rgb = inputs["image"]
    gt = inputs["depth"].clone()
    guidance = inputs[
        "depth_guidance"
    ]  # from GT if dense/synthetic or from a model's metric output
    validity_mask = inputs["validity_mask"].bool()
    dtype, device = gt.dtype, gt.device
    B, C, H, W = rgb.shape
    augmentable_indices = inputs["valid_camera"] & (
        inputs["depth_mask"].reshape(B, -1).float().mean(dim=1) > 0.0
    )

2. Regarding the splatting third-party library, when warping RGB images, is it also possible to warp the corresponding depth map? Does this involve converting the perpendicular depth into the radial depth?

I greatly appreciate your continued support and look forward to your insights.

Best regards,

[Gatsby23]

Dear lpiccinelli-eth:Thank you for your quick response.I have successfully created the camera_sampler based on your suggestions, but I have a few follow-up questions related to the augment_camera function:Could you kindly provide some guidance on how to properly organize the inputs for the augment_camera function? More specifically, I am unsure about how to handle inputs[depth_guidance]—how should this input be constructed and processed?Additionally, when working with the splatting third-party library to warp RGB images, is it also possible to warp the corresponding depth map? If so, does this process involve converting perpendicular depth into radial depth? If you could provide any insight into how this transformation is done, I would greatly appreciate it.Thank you once again for your continued support, and I look forward to hearing your thoughts on these questions.Best regards,Qi WuSJTU---- Replied Message ----FromLuigi ***@***.***>Date4/21/2025 ***@***.***>***@***.***>***@***.***>SubjectRe: [lpiccinelli-eth/UniK3D] Question about the camera_augment (Issue #7) Hi, thank you for your interest! I have not added it to the repo as I need to check some old commits of my dev repo, the exact camera sampler I used for the experiments in the paper. Anyway, I attached here one example that should resemble the one in the paper. I will get the original one and update the repo accordingly. Let me know if everything works fine or if any issues arise. camera_sampler.json—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***> lpiccinelli-eth left a comment (lpiccinelli-eth/UniK3D#7)Hi, thank you for your interest! I have not added it to the repo as I need to check some old commits of my dev repo, the exact camera sampler I used for the experiments in the paper. Anyway, I attached here one example that should resemble the one in the paper. I will get the original one and update the repo accordingly. Let me know if everything works fine or if any issues arise.camera_sampler.json—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[lpiccinelli-eth]

Regarding the followup questions:

1. input["depth_guidance"] is left on purpose "generic" because it is any dense depth that you have available, the "depth_guidance" is used to perform soft z-buffer when calling the external splatting function. If you have synthetic input depth, which is typically dense, ie valid for each pixel of the input, you can use that. If your ground truth depth is sparse, eg from LiDAR, Strcutured Light, SfM, etc, you would need to revert to some model's predictions. In our case, we "bootstrap" our own pinhole predictions in a no_grad context:

camera_sampler = CameraSampler()
with torch.no_grad():
    inputs["depth_guidance"] = model(inputs, image_metas)["depth"][:, 0]
    augment_camera(inputs, camera_sampler)
# rest of the code where you call the model with grad enabled

2. Yes, you can pass the depth to be warped, you can add anything you like in the depth_image variable here. The warping happens in the image plane, though guided by the "depth_guidance" to perform (soft) z-buffer check, so you should be able to pass any BCHW tensor whose elements will be moved according to the given flow variable.

Let me know if you have any further questions!