Timeframe: January 2023 - Present
Role: 3D Generalist + Environment Artist
Description: This project is an in-progress immersive learning simulation for semiconductor chip fabrication. The aim is to develop an experiential and competency-based approach that supports the development of a diverse and inclusive semiconductor educational ecosystem and skilled workforce throughout Ohio. This project is in collaboration between the Advanced Computing Center for the Arts and Design (ACCAD) and the OSU Nanotech West Lab. With the primary target audience being college students without much experience in a cleanroom environment, this partnership offers the necessary capacity to increase the representational diversity within the semiconductor industry, while simultaneously creating and piloting innovative strategies to engage in and expand upon the essential training and education of the future semiconductor workforce. Much of this VR experience is focused on Wet Bench Safety, including equipment operations and specific chemical procedures. Another focus area is the Cleanroom Orientation, including proper identification and use of personal protective equipment (PPE) and best practices for sample handling.
Stakeholders: College students coming from rural backgrounds who have not been exposed to a cleanroom before, senior-level STEM undergraduates preparing to do hands-on lab work
My Responsibilities: asset creation (3D modeling, texturing, retopology), assembling assets in a virtual cleanroom environment in Unity, collecting visual references, creating flat plane textures from reference images, 3D scanning lab equipment and optimizing it for use in VR (optimizing, sculpting, UV mapping, rigging) conducting the research needed for accurate and realistic representations of lab practices and user interactions, collaborating with developers and subject matter experts to brainstorm ideas and solutions for UX challenges
Core Team: Derrick Tillman-Kelly (College of Engineering), Rick Voithofer (College of Education and Human Ecology), Peggy Shadduck (Kent State University), Maria Palazzi (Design/ACCAD), Jeremy Patterson (ACCAD), Mila Gajic (ACCAD), John Carlin (Nanotech West Lab), Aimee Price (Nanotech West Lab), Dave Hollingshead (Nanotech West Lab), Tyler Grassman (Materials Science and Engineering), Siddarth Rajan (Electrical and Computer Engineering & Materials Science and Engineering)
Challenge:
Identify and utilize VR-specific opportunities to increase accessibility to safety training in cleanroom environments for undergraduate STEM students
Identify and utilize VR-specific opportunities to increase accessibility to safety training in cleanroom environments for undergraduate STEM students
Solution:
The VR-specific opportunities that this project is addressing include training situational awareness (e.g. being able to analyze and clean up a chemical spill), facilitating a safe exploration of processes and reactions (e.g. what happens when an acid and base are mixed), and also providing feedback mechanisms to instructors and students for enhanced training (e.g. being able to score 'performance' or in-situ feedback)
The VR-specific opportunities that this project is addressing include training situational awareness (e.g. being able to analyze and clean up a chemical spill), facilitating a safe exploration of processes and reactions (e.g. what happens when an acid and base are mixed), and also providing feedback mechanisms to instructors and students for enhanced training (e.g. being able to score 'performance' or in-situ feedback)
Project goals:
‣ Design the framework for and begin to build a comprehensive semiconductor curricular offering, including key experiential and hands-on learning within and beyond the classroom/lab, that are fully accessible regardless of educational background and geographic location
‣ Infuse simulated cleanroom practices and activities earlier in the curriculum (Engineering, Chemistry, etc.)
‣ Develop virtual replications of relevant instrumentation and provide simulated engagement to ready students for the use of physical cleanrooms and relevant instrumentation
The wet bench is the main interactive object in the virtual lab environment since the users get to interact with liquid simulations and chemical equipment on the bench, and they can use the aspirator wand and water and nitrogen guns the bench is equipped with.
Wet Bench Safety module - learning objectives:
‣ Identification of hood components and basic operation
‣ Proper identification of chemicals/materials
‣ Proper pouring of chemicals and basic sample use
‣ Proper waste disposal and cleanup
Other high-detail objects (spin rinse dryer and emergency eyewash station) - modeled and placed to provide a more realistic impression and experience within the virtual lab, even though they are not interactive in VR:
My main contributions to the project involved 3D modeling, texturing, and optimizing all assets needed for the virtual cleanroom environment. Upon collecting photo references and 3D scanning the room layouts, I began modeling assets, starting with high-priority ones like the wet bench. I used Autodesk Maya for modeling, Substance Painter for texturing, and Substance Stager for exporting the textured models as GLBs, to later use some of them for AR visualizations that we compared to the actual objects in the Nanotech West Lab to assess whether the essential features like the scale and functionality are compatible. The five images below show the first iteration of the wet bench and emergency eyewash station compared side to side to the actual objects using AR, which helped identify the modifications I needed to make such as lowering the top part of the wet bench and making the eyewash taller. After polishing individual assets, I imported and assembled them in the virtual lab environment in Unity.
Augmented Reality-based comparisons of the first iteration of the wet bench and eyewash models with actual equipment at the OSU Nanotech West Lab:
Real wet bench on the left - virtual model next to it on the right
Standing next to the real wet bench with the virtual model behind me
Real eyewash station on the left - virtual model next to it on the right
3D scanning and optimization:
In addition to modeling environment assets, I also 3D scanned multiple versions of the lab clean suit on a mannequin. This task involved retopologizing the original mesh in ZBrush so that it can be used efficiently on animated 3D avatars in the virtual environment. I also created new UV sets to which I transferred the original suit texture in Maya:
In addition to modeling environment assets, I also 3D scanned multiple versions of the lab clean suit on a mannequin. This task involved retopologizing the original mesh in ZBrush so that it can be used efficiently on animated 3D avatars in the virtual environment. I also created new UV sets to which I transferred the original suit texture in Maya:
Comparison of the original versus optimized meshes and UV texture maps:
Cleanup and retopology of the clean suit mesh
Original UV texture map of the 3D scanned mesh on the left - optimized UV texture map on the right
I repeated the same process of 3D scanning, retopologizing and UV mapping for the lab gloves which we later used as custom VR hands that students perform the wet bench tasks with. I used the polished glove mesh to bind it to the default Oculus hand rig. I then mirrored the same glove mesh in order to get a left glove as well, and I repeated the same process of applying the Oculus hand rig and painting the skin weights in Maya. I then replaced both hand meshes in Unity with the custom lab gloves containing the same rig as the default Oculus hands:
Cleanup and retopology of the 3D scanned lab glove in Maya
Gloves ready for import to Unity
Using the gloves in VR
Virtual cleanroom environment (in progress):
The goals for this VR experience are to focus on training on the concepts needed for safe and effective lab use, build good habits, and prioritize attention to detail. The virtualization workflow avoids replicating unnecessary and time-consuming details but focuses on equipment specifications:
The goals for this VR experience are to focus on training on the concepts needed for safe and effective lab use, build good habits, and prioritize attention to detail. The virtualization workflow avoids replicating unnecessary and time-consuming details but focuses on equipment specifications:
Shaded wireframe mode in Unity - PPE room:
Wireframe view of the clean suits in the VR space
Entrance to the lab
Room where PPE is donned before going into the cleanroom
After preparing the virtualized clean suit for VR and AR, I used Character Creator 3 to equip the clean suit onto various human avatars. After painting skin weights to prepare characters for animation, I used iClone and Mixamo to create simple idle, working, and sitting animations to import into the virtual environment in Unity:
Virtual avatar working in the VR lab
Avatar in the VR lab
Animated virtual avatar in the PPE room
Asset creation was also influenced by getting some basic firsthand lab experience thanks to the Nanotech West Lab staff. This experience helped me understand what parts needed to be interactive and more detailed than others.
Visiting the OSU Nanotech West Lab to observe and engage with common lab practices and collect photo references for virtual objects:
Visiting the OSU Nanotech West Lab to observe and engage with common lab practices and collect photo references for virtual objects:
Testing the early iteration of the project with fundamental wet bench interactions such as functioning water and nitrogen guns and a functioning aspirator that interacts with an in-progress liquid simulation:
