Designing machine learning app that helps to create ML models from any object
ML model creator app
How do you make machine learning easy to use and apply for business purposes?
OVERVIEW
Research, Information Architecture, Feature prototyping, UX, Visual design & testing
Role: Key Designer | EPAM Systems
Team of 7 developers
January 2021 - October 2021
Main challenge of this project was to build a system, allowing to create data models of any real object using a regular mobile device.
PHASE 1
Discovery and planning
Discovery and planning
I've started with discovery, understanding the need and benchmarking. Having access to potential users - I've conducted series of interview to better understand their current pains, needs and preferable workflow. After 8 interviews with end-users and 2 workshops with stakeholders and development representatives I managed to propose an MVP feature list.
Findings and insights
87% are not familiar with Machine Learning (ML)
People tend to hear about machine learning, but were not sure about how it works. Which gave me a strong feeling that proper onboarding for the app will be needed.
43% used a 3D modelling software at least once
Understanding the volume of real objects and being able to interact with them was one of the important things for the project to succeed. Getting mixed responses from user interview gave me an insight of introducing several tools to create a 3D "bounding boxes" - simple way and advanced.
78% of potential users happen to be technicians
Our target audience mostly consisted of technicians who were eager to make their workflows more efficient and were looking forward to incorporate modern technologies into their daily tasks.
PHASE 2
User flows
User flows
I've started with discovery, understanding the need and benchmarking. Having access to potential users - I've conducted series of interview to better understand their current pains, needs and preferable workflow.
PHASE 3
Tools behavior research
Tools behavior research
How to make process of bounding box creation easy, keeping in mind technical limitations?
Important part of data model creation is annotating the real object. To do so you need to mark areas (bounding boxes) of an object and give it a proper name so that machine learning algorithm will be able to make sense out of this data.
I've introduced several tools to help with this task. Most of them were aimed at dealing with complex 3d shapes. However, after quick user tests I've realised that we were still missing an easy and quick way to place primitive volumes. Having pre-made templates could help, but I've decided to go further and came up with Freehand tool.
I've introduced several tools to help with this task. Most of them were aimed at dealing with complex 3d shapes. However, after quick user tests I've realised that we were still missing an easy and quick way to place primitive volumes. Having pre-made templates could help, but I've decided to go further and came up with Freehand tool.
Move & scale tool
Allowing to move bounding box along X,Y,Z axes
by dragging cube using arrows. Scaling is intuitive - using zoom in and out finger gestures.
by dragging cube using arrows. Scaling is intuitive - using zoom in and out finger gestures.
Adjust tool
Allowing to extrude surfaces by selection and dragging into needed direction
Vertices repositioning
Allowing to change vertices position in X,Y,Z space by simply dragging them. This tool opened possibilities to create complex and more organic shapes.
Rotate tool
Allowing to rotate bounding box with a step of 10 degrees at any direction.
Freehand
This tool was meant to solve the issue of bounding box
creation for those users, who never used 3D modelling software before. Simple "draw and convert" concept
was introduced.
creation for those users, who never used 3D modelling software before. Simple "draw and convert" concept
was introduced.
PHASE 4
High fidelity prototyping
High fidelity prototyping
PHASE 5
Making things real
Making things real
In case you're an active user you'll have a chance to open any of existing projects straight away.
Starting the journey with loading the app and creating new project
Here user can access any of created projects or start a new one. Each project can contain as many products as user likes. And each product can be at different state, depending on whether it was already trained or still waiting that to happen.
Not trained yet
In training
Trained
Trained but unloaded (so the edit is not possible)
Projects overview screen
Labeling session is a term we call the process of marking down areas of the object. This step is crucial for creating good recognition results after data model will be created.
Labeling session
User starts with placing a 3D cube or a freehand drawn shape on a canvas
Adjusting the shape to match with objects part you want to annotate
Fine tuning to get the perfect shape via vertices adjustment is also available
As soon as bounding box is placed - time to name the label, so that it can be saved as a part of future data model.
Naming and saving a label
You can always access all labels on scene, delete it or change its name or bounding box shape
Accessing all labels on a scene
During scanning user is asked to walk around an object while the app is taking series of pictures which will be later used as a base for data model creation and machine learning.
To help user understand when it is enough to film specific surface of the bounding box we've introduced dynamic color filling, which visually shows which area is already covered good enough.
To help user understand when it is enough to film specific surface of the bounding box we've introduced dynamic color filling, which visually shows which area is already covered good enough.
Scanning flow
Finally you can start a data model training process by selecting scans you've just did. System will update you on a status of each data model and will notify when training will be done.
Data model training
© 2023. Darya Tarasevich