Synthetic Data Generation for One-shot/Few-shot Image Synthesis: A Comprehensive Approach

Abstract

The generation of synthetic data has become an essential tool for machine learning, especially in areas such as computer vision, where the collection of real-world data is often expensive, time-consuming, and limited by practical constraints. This paper presents a Synthetic Data Generator developed using Unity 3D, which enables the creation of diverse and realistic datasets for one-shot and few-shot learning applications. Unity’s robust rendering engine allows for high-quality image synthesis, while custom scripts written in C# handle the configuration of various randomization techniques.

The generator incorporates randomization of key image features, including surface roughness, lighting conditions, camera angles, object movement, and backgrounds. These randomization features simulate real-world scenarios and generate data that mimics the variability encountered in real-life environments. On the backend, the system supports parallel data generation, ensuring scalability for large datasets. Its cross-platform nature enables deployment on Mac, Windows, and Linux.

1. Introduction

1.1 Background

In fields like autonomous driving, robotics, and healthcare, machine-learning models require significant amounts of labeled data to achieve robust performance. Real-world data collection is costly, time-consuming, and constrained by ethical or logistical issues — HIPAA and GDPR restrict patient data in the medical domain, and rare edge cases (unusual road conditions, uncommon diseases) are difficult to capture in real life but crucial for training. Synthetic data addresses these limitations directly: scalability, accessibility, and diversity, all without the cost of real-world capture.

1.2 Motivation

The motivation is the need for customizable, high-quality image datasets that enable effective one-shot and few-shot learning. Traditional image synthesis workflows lack variety, realism, and flexibility over critical parameters such as lighting, texture, movement, and camera angle. This generator introduces randomization techniques over each of these to produce datasets that closely mimic the complexity of real-world environments.

1.3 Problem statement

Most available synthetic data generators focus on basic randomization — limiting their ability to represent complex scenarios. They also struggle with scalability: the compute required for high-resolution images or complex environments becomes prohibitive. This generator addresses both by offering advanced randomization, backend support for parallel processing, and a flexible export system.

1.4 Contributions

1. Advanced randomization features: texture roughness, lighting, camera angles, and object movements can all be varied per frame.
2. Scalable backend architecture: parallel processing enables efficient generation across clusters.
3. Cross-platform usability: the generator runs on Mac, Windows, and Linux.
4. Future directions: multi-object detection, semantic segmentation, and cloud deployment via AWS EC2.

2. Methodology

2.1 System overview

The generator allows user-uploaded images to be applied as textures to specific sections of 3D objects in a scene. Rather than covering the entire object with the texture, the system first segments the object — separating the region where the image will be applied — so only a designated portion is affected while the rest retains its original appearance. Segmentation happens at the mesh level; a C# script then assigns the uploaded texture to those specific meshes.

2.2.1 Object segmentation and texture application

For example, if the object is a car, the system can segment the door or the hood and apply the uploaded image only to that part while leaving the rest of the car intact. The remaining parts retain their default textures, keeping the object visually cohesive.

// Get the renderer component of the segmented part
Renderer partRenderer = segmentedPart.GetComponent<Renderer>();
Texture2D uploadedTexture = Resources.Load<Texture2D>("UploadedImage");

// Apply the uploaded image to the specific section of the object
Material partMaterial = partRenderer.material;
partMaterial.mainTexture = uploadedTexture;

2.2.2 Surface roughness and material randomization

After the uploaded image is applied, glossiness and reflectivity are randomized to simulate different material types — from glossy, reflective surfaces to rough, matte textures.

// Adjust the glossiness of the material on the segmented part
partMaterial.SetFloat("_Glossiness", Random.Range(0.1f, 0.9f));

2.2.3 Lighting conditions

Both natural lighting (sunlight) and artificial lighting sources within the scene are randomized. These changes affect how the applied texture is perceived, enhancing realism by simulating different lighting environments.

Light sceneLight = GameObject.Find("SceneLight").GetComponent<Light>();
sceneLight.intensity = Random.Range(0.5f, 2.0f);
sceneLight.color = new Color(Random.value, Random.value, Random.value);

2.2.4 Camera angle and movement

Camera randomization captures the object from different angles and perspectives. By changing the camera’s position and rotation, the system ensures the segmented object with the uploaded image is viewed from many viewpoints.

Camera.main.transform.position = new Vector3(
    Random.Range(-10.0f, 10.0f),
    Random.Range(1.0f, 5.0f),
    Random.Range(-10.0f, 10.0f)
);
Camera.main.transform.LookAt(object.transform);

2.2.5 Object movement and rotation

The object itself can be randomized in position and rotation. Random forces or angular velocities are applied so each render captures the object in a different pose — particularly useful when the object is a car, a robot, or any subject that moves in real-world data.

Rigidbody objectRb = segmentedPart.GetComponent<Rigidbody>();
objectRb.velocity = new Vector3(Random.Range(-2.0f, 2.0f), 0, Random.Range(-2.0f, 2.0f));
objectRb.angularVelocity = new Vector3(0, Random.Range(-1.0f, 1.0f), 0);

2.3 Backend architecture — parallel processing

Unity’s coroutine-based multi-threading enables parallel processing, so multiple images can be generated simultaneously. The generator can produce large datasets efficiently, even when applying complex textures and randomizations.

StartCoroutine(GenerateImages());

IEnumerator GenerateImages() {
    for (int i = 0; i < numberOfImages; i++) {
        CreateScene();
        yield return null; // Proceed to the next frame
    }
}

2.3.2 Export functionality

Rendered frames are exported in PNG or JPEG, along with metadata that records the randomization parameters for each image — giving the training pipeline a replayable audit trail.

ScreenCapture.CaptureScreenshot("image_" + i + ".png");

3. Results

The generator was tested with user-uploaded images and pre-defined 3D models. Batches of 1,000 images were generated per configuration, each capturing a unique variation in texture placement, lighting, camera angle, and pose. On average, the system produced approximately 100 images per minute, depending on model complexity and texture resolution.

Scalability was tested against datasets from 100 images up to 10,000 images. In all cases, performance remained consistent, with no significant slowdowns or bottlenecks. Exported images integrated cleanly into standard ML pipelines — each file was paired with metadata describing the randomization parameters used during generation, making it easy to reproduce specific conditions during training.

4. Discussion and future work

Randomization drives dataset diversity. Varied lighting, camera angles, object movement, and material properties ensure that each generated image captures the object from a different perspective or in a different environment. This is particularly important for object detection, where models need to recognize objects regardless of orientation.

The system is well-suited to autonomous driving, robotics, industrial inspection, and medical imaging. Planned improvements include:

1. Advanced object segmentation. Better handling of intricate or detailed models without manual intervention.
2. Physics-based rendering. More realistic material response to lighting and inter-object physical interaction.
3. Multi-object scenes. Rendering multiple objects interacting in the same scene for more comprehensive datasets.
4. Cloud-based deployment. Distributing generation across AWS EC2 or Google Cloud servers for larger datasets in less time.
5. Semantic segmentation and labeled data. Automatic pixel-level class labels alongside the rendered image.

5. Conclusion

The generator successfully applies textures to segmented parts of objects while maintaining visual cohesion. Randomization of lighting, camera angles, and object movement produces datasets that are rich in variation, and Unity’s parallel processing keeps generation fast enough for both academic research and industrial applications. With planned enhancements in segmentation, physics-based rendering, multi-object scenes, and cloud deployment, the tool is positioned to make meaningful contributions to autonomous driving, robotics, industrial inspection, and medical imaging.

References

1. Unity Technologies. Accelerating Computer Vision Training with Synthetic Data. unity.com
2. Sankaranarayanan, S., Balaji, Y., & Chellappa, R. (2023). Machine Learning for Synthetic Data Generation: A Review. arXiv:2302.04062
3. Goyal, M., & Mahmoud, Q. H. (2024). A Systematic Review of Synthetic Data Generation Techniques Using Generative AI. Electronics, 13(17), 3509. doi.org/10.3390/electronics13173509