Initial results of using Hunyuan3D V2 image-to-3D AI. The results in this section are made in ComfyUI.
By using FLUX Kontext, we can edit the source image to change the output with a prompt:
Change the pot in the photo to be a futuristic design. Keep the proportions the same.
Create a geometric stone sculpture that has similar features to the artifact in the photo.
A variant of Hunyuan3D-V2 has the ability to take multiple images as input to help generate the 3D mesh. It was trained on orthographic views of the input object, which requires aligned images, consistant lighting and clear backgrounds to work fully.
The Ai Khanoum Sundial is a stone sculpture that tells time on a hemispehrical surface cut out of a large block of sandstone, with lion paw details. From combining the images avaliable, and pushing the models resolution to its maximum value, this is the result:
The model is scaled to have a 52x46x47cm bounding box, as described in the Harvard article. There are some details such as the lions paws, however it lacks the cracks and the iron bars supporting the broken structure, along with the veins on the lion legs. Below is a comparison between the mesh and drawings from Serge Veuve (orange lines are the mesh):
Some parts are not quite aligned, such as the cutout in the front towards the hemisphere, and the amount the lions legs protrude. This would be relatively easy to adjust and fix in 3D software. It also seems that simply re-generating the mesh will yield slightly different results, so multiple can be created and the most accurate model selected.
However, it seems this method of generating the mesh is not suitable to extract all of the fine details of the sundial, such as cracks on the side. A proposed solution is to use normal maps to add these details. This is a method in computer graphics to add surface details using textures, rather than adding detail to the 3D mesh itself. Given the detailed sketches of the top and sides of the sundial, I used AI to generate an image of a stone with the same shape and details as in the sketches. From these images, a normal map was extracted using DeepBump, an algorithm to generate normal maps from images of textures.
These can be applied to the 3 known sides of the sundial mesh to add the extra detail. coming soon
However, the most accurate result was obtained using a single image of the sundial from a 3/4 view (same as in the first section).
The resulting mesh was adjusted in Blender using sculpt mode and proportional editing to make the bowl more spherical and align certain features to the cross-sectional view of the sundial:
A note of size: the reported dimensions (52cm x 46cm x 37cm, width depth height) do not quite match up with the proportions in the sketches. Potential reasons for this could be the reported size if on a different, but similar, sundial. The sundial may also have incurred some damage - in the photo we cannot see the three holes in the cutout and the back wall looks thinner than in the sketches. After scaling to the reference images, the model is closer to 47cm x 46cm x 41cm.
Update: After revisiting the sketches it was found that at some point in the digitisation of the report, the images were distorted slightly. After correcting this by stretching the images to make the sphere perfectly round, the proportions of the image matched the described dimensions. The top-view sketch had to be stretched 3% wider, and the side view drawings 10%.
A note on sizes: all the objects generated below are the same dimensions, namely they are scaled to fit in a box 100x100x100cm. This is the default size from the output of Hunyuan3D. With no other clues or context, it is up to the artist to scale the objects to the final dimensions.
Some of the images in the Kabul Museum objects are very old and of low quality: black and white scans of old photographs, with a lot of grain and over- and under-exposed areas that lose much of the details. We can use FLUX-Kontext to restore some information and generate a (speculative) reconstructed image of the object to generate a cleaner output:
No image restoration:
With image "restoration":
"place the old vase in the photo on a table in a bright room.
With more context in the image, the model can better recognise the object as a vase. However, it still does not do well with the repeated honeycomb pattern on the surface of the object.
With the techniques described in Disrupting Model Pipeline below, we can use a clear, generated image of a similar shaped object to help the model to generate something that is closer to a typical vase:
The shallow patterns on the surface of the objects are not well recovered by Hunyuan3D, no matter the parameters chosen (octree resolution and number of inference steps have dimishing returns while bumping them up).
Even with a very "clean" reimagined version of the beaker, it is clear that Hunyuan3D model is simply not capable of recreating the indented patterns on the surface with any accuracy:
Ideas to bring back surface details:
It is unclear on what data Hunyuan3D and others are trained on, but looking at the examples given by the developers and tutorials, it seems to be more optimised for cartoon rendering, rather than realism. As one of the model features is "text-to-3D", the mesh generation is likely also best at rendering AI generated images (which is the intermediate step in text-to-3D).
In an attempt to bring back the surface details, first a 'blank' object is generated. This object has a smooth surface with the idea to add details iterativly later:
Next, we use Hunyuan3D's Texgen to create an image texture that wraps the object. This way we can use the image to extract more surface details to apply to the mesh.
Next, we use a tool that extracts surface details from the texture image (DeepBump). The first image is the normal map (colours encode the surface direction) and the second is a height map (brighter areas are further from the surface). It can be seen that some of the depressions are captured by this method.
However, this process yields mixed results, with only some semblance of the surface texture recreated. A lot of clean-up is required in areas and at the texture seam, so much that it might be better to manually sculpt the beaker from the blank model above.
There are various models that try to understand the depth of an image and produce a "depth map" that estimates the distance of every point in a still image from the camera. We could use this to estimate the relief pattern on the surface and recreate it on the object.
Using Depth Anything V2 model, the following was generated:
There is some potential with this method, as the depth estimation found the small indentations on the surface, but it would require a lot of work to map this information onto the body of the blank model. The resolution is also limited, in both vertex density and depth range.
After many attampts, I managed to get a "restored" image of the beaker that maintained all the features of the original image. This same image was inputed four times into Hunyuan3D as four "different" angles of the object to give it the best change to recreate the pattern around the whole object.
For everything that was tried, it seems very unlikely that the models can reproduce the relief pattern accurately all the way around the beaker. The results shown above were the best after a lot of trial and error and failed attempts.
I think the best way to reproduce this object accurately would be to manually sculpt the "blank" mesh. It should be quite straight forward to create a height map manually that mimics the pattern and apply it to the model.
After restoring the image of the glass beaker, Hunyuan3D fails to recognise that the glass caraf in the image is infact broken and missing most of one of the sides. We can try to use the texturing technique from the small beaker to outline where on the caraf the wall is missing, and manually delete the vertices from the mesh itself.
The vertices of the model that lie on the regions of the texture that represent the void (darker regions) were deleted in Blender. A small thinkness was added to the now hollow caraf. Note: the hole at the top does not yet go all the way through! This can be easily fixed manually.
Generating the fish sculpture is quite straight forward, and improved by restoring the image first. Here, I also remove the supports in the image. As there is a lot of bluriness there is a lot of room for the AI to interpret the shapes of the fins and tail.
Restore and colourize the photo of a fish sculpture. Remove any scratches and sharpen the image. Remove the two supports. Maintain the details.
Initial results failed to realise that the two triangle features at the top are parts of the flaired opening that have been broken (see the first result at the top of the page). To remedy this, we try the technique of Conditional Averaging outlined below to mix in an image of a similar shaped object that has the correct style of opening.* The image of the wavy glass caraf was also restored using FLUX-Kontext.
* The prompt used was: "Create an image of a carafe with the same teardrop shape and wide opening as the object in the photo. Remove the ouside details but preserve the proportions of the interior shape", with the image of the wavy caraf as additional context.
It can be seen that a small percentage of the artificial caraf mixed into the archival images produces a cleaner object with a more accurate opening at the top.
Note: Hunyuan3D seems to avoid making any object like the caraf hollow with a hole at the top (e.g. in the artifical caraf). This can easily be acheived with some manual editing.
By exposing the compenents and processing steps in the Hunyuan3D image-to-3D AI, we can manipulate the data at various points during the generation. With this technique, we can condition the model on a higher quality (potentially generated) image of the object that has roughly the same shape and size as the target. For example, if the model does not recognise the input as a vase, we can help it out by combining it with a similar, clearer image of a vase and do some interpolation to find the balance between the target and expected output. The standard pipeline of the model is shown here:
┌ ─ Hunyuan3D─V2 ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐
│ Sigmas ────────┐ │
│
│ Condition + Timesteps ──────┤ Latents │
┌───────┐ Encode Input │ Sample Model to Mesh ┌─────────┐
│ IMAGE ├────┼───────────────▶ Condition ──────●═══════════════▶ Latents ──────────┼───▶│ 3D MESH │
└───────┘ │ └─────────┘
│ Latents ────────┤ │
│
│ Guidance ───────┘ │
└ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘
[source]
This is the regular flow of Hunyuan3D, where the input image is processed into a condition which is an abstract represention of the features of the object. A few more parameters are created by the model (sigmas, timesteps, guidance) that are important for the next step. The latents vectors are initialised with pure white noise, which is essentially the 'canvas' the model works with to 'draw' the 3D object.
All of these are used by the model in an iterative process ("sampling") which converts the noisy latents into a structured representation of the final 3D mesh. This new latents is then interpretted by another model to transform it into the 3D mesh.
Blend between the latent representation of the two 3D objects before converting to model.
┌ ─ Hunyuan3D─V2 ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐
│ Condition + Sample │
┌─────────┐ Encode Input Model
│ IMAGE 1 ├────┼───────────────▶ Condition ──────●══════════▶ Latents 1 ───┐ │
└─────────┘ │ │
│ │ │ │
Latents ────────┤ │
│ │ │ Latents │
Guidance ───────┤ ▼ to Mesh ┌─────────┐
│ │ Weighted Average ────────────┼──▶│ 3D MESH │
Timesteps ──────┤ ▲ └─────────┘
│ │ │ │
Sigmas ────────┤ │
│ Condition + │ │ │
┌─────────┐ Encode Input │ │
│ IMAGE 2 ├────┼───────────────▶ Condition ──────●═══════════▶ Latents 2 ───┘ │
└─────────┘ Sample
│ Model │
└ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘
Move the slider to interpolation between the two objects.
Blend the conditioning vectors from the input images before generating the 3D models.
┌ ─ Hunyuan3D─V2 ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐
│ Condition + │
┌─────────┐ Encode Input
│ IMAGE 1 ├────┼───────────────▶ Condition 1 ───┐ │
└─────────┘ │
│ │ Latents ────────┐ │
│ │
│ │ Guidance ───────┤ Sample Latents │
▼ │ Model to Mesh ┌─────────┐
│ Weighted Average ───────────────●══════════▶ Latents ──────────┼─▶│ 3D MESH │
▲ │ └─────────┘
│ │ Timesteps ──────┤ │
│ │
│ Condition + │ Sigmas ────────┘ │
┌─────────┐ Encode Input │
│ IMAGE 2 ├────┼───────────────▶ Condition 2 ───┘ │
└─────────┘
│ │
└ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘
Move the slider to interpolation between the two objects.
