Erase.bg: Instant Background Removal for Everyone

This blog is intended to introduce the reader to the world of background removal and share some of the learnings and challenges that came our way when creating Erase.bg. Erase.bg is an entirely automated background service that offers high-resolution outputs for free. So Let’s Get Started!

‍

Deep Dive into Background Removal

‍

Background removal is selecting the foreground subject in an image and erasing the background so the foreground subject can be placed on a new background. This is done with the help of a mask produced in the removal process, as shown below. Placing the subject on a new background is called composition.

‍

As the human eye is susceptible to discrepancies in images, great care must be taken so that the composition of the extracted subject looks realistic on a new background. This can be tricky for delicate structures like hair and fur.

‍

Composition allows us to reuse images by putting them on formal or creative backgrounds. There are two significant challenges in creating a completely automated system that can do background removal:

‍

1. Detecting the foreground subject without human input: As the foreground subject can belong to a diverse range of objects like humans, animals, electronics, clothing, furniture etc., our model must identify and handle the subject even if our model has never seen that subject before. Handling interactions with known subjects, like a human playing basketball or sitting on a chair, can also be tricky.

‍

2. Predicting an accurate mask: The mask must cover only the subject area. Moreover, many structures like fur and human hair have delicate semi-transparent and intricate structures that must be handled with care to ensure precise extraction and realistic composition.

‍

Real Applications of Background Removal

‍

In the next section, we will elaborate more on these challenges and show examples to explain them in detail. For now, let’s look at some applications of background removal:

‍

1. Reuse photoshoots: Get more out of your existing content

‍

2. Ecommerce: Get your product images ready and make them look more appealing for any E-Commerce platform

3. Media/Creative: Unfold your creative edge and make stunning content

‍

4. Profile and Passport Pictures: Create stunning profile pictures in just a few clicks. Skip the line and create passport photos anywhere.

‍

5. Logos/Signature/Graphics: Promote your business or any of your cool projects

‍

Challenges

Our goal is to automatically produce a mask to extract the foreground object from the image. There are two main challenges in doing this:

‍

1. She is automatically detecting the foreground subject.

2. Predicting an accurate mask for realistic composition

‍

While detecting the foreground object is easy for a human, it is not so for a computer. We focus on the second challenge in this post and leave this one for a future blog post.

‍

A common approach to generating a mask is a technique called image segmentation, which assigns a binary value, a YES or a NO, to every pixel in an image to indicate if that pixel belongs to the subject. Then, this mask can be used to extract the object, as shown below.

‍

We can see that image segmentation did the job perfectly here. However, in more complex cases, such as those with delicate hairs, image segmentation will not do the trick, as seen below. Even though the mask covers the hair pixels, something is not right in the extraction. This is because part of the subject, particularly her hair, is semi-transparent at certain places and interacts with the lighting and color of the background to produce the soft tone we see in the original image.

‍

In such a case, what we need is a transparency mask or alpha matte for proper extraction, as shown below. Such a mask can completely separate foreground and background elements, even at places where they deeply interact.

‍

The created masks must have a high level of accuracy as compositions created from background removal are often used for eCommerce or media purposes where humans would be seeing them. Let's talk about three factors that impact the realism of compositions:

‍

• Color decontamination

• Small gaps

• Alpha blending

‍

Color decontamination

‍

Consider the below example to illustrate how slightly different masks can cause significant visual differences in composition. The top image is the original image which has a stuffed toy on a bright blue background. Shown below it in the first column are two masks generated by two different algorithms. The masks look very similar; however, when the extraction is done using the first, the blue light is still partially visible on the subject's hair, as shown in the second column.

‍

In the third column, we can see close-ups of a small patch on the right-hand side of the toy. Here, we notice that the first algorithm has slightly overestimated its values leading to more of the background light being taken in the extraction. Both cover almost the same hair area, yet their precise values make a big difference.

‍

Algorithms must understand the colours and lighting interactions of the foreground and background to circumvent such issues. Even a human would find it difficult and time-consuming to produce such a mask himself, and that's where background removal comes to the rescue.

‍

Small gaps

‍

Even small errors in the mask, such as missing tiny gaps, can cause obvious visual discrepancies when composing on a new background. Below we see the original image on the top and masks produced by two algorithms on the right.

‍
A range of errors in the first mask is visible. Large gaps like the one near the shoulder are clearly visible when the subject is placed on the green background, but so are the tiny ones around and within the curly hair. If these gaps are left out, then parts of the old background are placed on the new one making the image look synthetic.

‍

Alpha blending

‍

Let's see how our transparency mask promotes realistic composition. Our transparency mask indicates how much of the foreground and background colours should be present for every pixel.

‍

For example, at a certain pixel, we could have a blend of 60% foreground colour and 40% background colour. Such blending makes it appear as if the subject was truly present in the new background and interacting with the lightning there.

‍

Compositions also seem less jaggy around the edges and more natural and appealing to the human eye. When composing, we don't want the background to start abruptly.

‍

We want a smooth transition from the foreground subject to the background region. To illustrate, we compose a small patch of brown hair extracted from the corner of the stuffed toy above in c) on the bright background in a) below.

‍

When we blend with the help of our mask in b), we see a smooth transition and interaction of colours in the composition produced in d), making it appear as if the subject was actually present and interacting with the lightning of the new background. There are more advanced techniques for composing subjects and backgrounds from different lighting conditions, but we do not discuss them here.

‍

In conclusion, we saw that background removal is a delicate process that requires transparency prediction to get it right. Using the map generated in the removal process, we are able to extract the subject and compose a new background.

‍

We discussed three important considerations that affect the realism of the compositions and the level of accuracy and detail expected from algorithms that attempt to produce these masks. To accomplish this, we must rely on advanced machine learning and computer vision methods that we will cover in a future post.

‍

We hope you enjoyed reading this blog and now have a deeper appreciation of background removal and its many applications. To see all this in action, try out our background remover Erase.bg which is set to launch with a new website soon. For more updates, follow us here.