Question about object insertion

[bring728]

Hello, Thank you for sharing your amazing research. I am currently attempting object insertion using x2rgb, but the results seem a bit off, and I’d like to understand what might be going wrong.

this is the input img.

First, I ran RGB -> X for the scene, and this worked without any issues.

Next, I performed editing using the material and normal information of the object I want to insert, as shown below. I didn’t modify the irradiance in this step.

Afterward, I used your model for inference with the following settings:

seed=2256
inference_step=10
guidance_scale=7.5
image_guidance_scale=1.5
However, I got some strange results, as shown below.

Could you help me understand what I might be doing wrong?

[bring728]

It seems that not including shading produces better results. When shading is included, it appears that shadows cannot be rendered properly.

Is there anything else I should be mindful of?

[zheng95z]

Hi @bring728,

For object insertion, the steps we take are:

• Run RGB->X to get background intrinsics (albedo, normal, roughness, metallicity, and irradiance).
• Drop irradiance -- we will not use this in the following steps.
• Perform insertion to get new intrinsics new_* -- put objects intrinsics onto background intrinsics.
• Run X->RGB-inpainting with original background image, mask (masking the inserted objects), new_albedo, new_normal, new_roughness, new_metallicity as inputs.
• (Optional) Use a second mask to clean up artifacts in the transition area.

Best,

[bring728]

Thank you for your quick and kind reply!