Create AI x AR Instagram filter using Midjourney & Spark Studio

Kiki Wu

AI Artist

Visual Artist

AR/VR Developer

Instagram

Midjourney

Spark AR Studio

The impact of Augmented Reality (AR) on modern technology has been significant, with many people immediately thinking of the 2016 launch of the Pokémon Go game. Created by Niantic Studios, Pokémon Go made AR technology mainstream and showcased the technology's ability to merge virtual gaming experiences with reality. By doing so, AR allows creators to bring about a more engaging way of storytelling.

Interactive Storytelling with AR

If I had to describe AR straightforwardly, I would say AR superimposes digital content on top of the real world.

One of the most enjoyable AR experiences I've had is creating historical AR artwork for San Jose Japantown. The "Hidden History of San Jose Japantown" experimental AR Project aims to connect locals with the neighborhood's beauty and rich history through a historical walking tour featuring AR artworks.

Placing location-based augmented reality artworks in historical locations can help people uncover the hidden significance of the neighborhood. What's intriguing is that even if certain buildings are no longer present, the memory of them remains in the same spot.

AR artwork map from Hidden History of San Jose Japantown Website

AR Photo Filters Add More Fun to Life

AR technology differs from Virtual Reality (VR) in that it expands the potential of our real world rather than transporting individuals to a completely different environment.

AR adds a new layer of possibilities to users' daily lives that are accessible through their smartphones. Another example of the increased accessibility of AR is demonstrated by photo filters, provided by many social media platforms like Instagram, Snapchat, and TikTok.

These AR effects enable users to change their appearance in real-time using makeup filters or face masks featuring animal ears during selfies and video calls, resulting in more fun social experiences and connecting people in humorous ways.

Types of AR

In addition to real-time photo filters and location-based AR like Pokémon Go and Hidden History of San Jose Japantown project, there are several other AR experiences that rely on specific triggers:

Marker-based AR: This type of AR uses specific physical images, such as markers, captured by the camera to position digital content on top of them. Markers can be images, logos, posters, or QR codes. For example, the below video is an AR artwork based on the image maker, which is my passport.

View post on Instagram

Location-based AR: This type of AR ties the augmentation to a specific location by using real-time data from a smartphone's camera, GPS, digital compass, and accelerometer. Pokémon Go is one of the most significant location-based AR games.

Projection-based AR: This type of AR projects digital images onto physical objects in the real world. For example, it can project an interactive digital keyboard on your desk or a dialer on your hand.

Superimposition-based AR: This type of AR detects objects in the real world and partially or fully replaces their original view.

Face Recognition AR: This type of AR utilizes facial recognition technology to track and map facial features. AR filters can be applied to a user's face in real-time, which is commonly used in social media platforms like Snapchat, Instagram, and TikTok.

AI x XR

In my recent tutorials about integrating AI-generated images into immersive artworks, I take pleasure in trying out and merging novel technologies.

This has inspired me to explore the possibilities of combining AI art with XR technology, resulting in some exciting experiments that bring AI art to AR effects.

In this article, I will be concentrating on developing an exciting photo filter using Midjourney and Meta Spark AR Studio.

A step-by-step guide to creating AI x AR photo filter

Prepare your tools

Midjourney: Midjourney is the most popular AI image generator. I use Midjourney to create image assets for AR materials

Meta Spark AR Studio: Spark AR studio is an asocial AR effect builder that allows you to create different types of AR effects for Instagram. including face masks, 3D posters, AR environment… etc.

Adobe Express Remove Background: Say goodbye to tedious background removal with the Adobe photoshop laser tool! Simply upload your image to Adobe Express and it will automatically remove the background for you.

Adobe Photoshop: I use Adobe Photoshop to modify my AI images and map them onto the faceMeshTracker template.

Step 1: Generate face masks and decorations using Midjourney

To create the material for the AR face filter, I generate several face mask images by typing words like face mask, and eye mask.

Feel free to add any style words like fantasy, or sci-fi.

Meanwhile, I also generate some decorations so I can remix them in Photoshop.

Step 2: Remove the background using Adobe Express

Gather your AI images and upload them to Adobe Express to remove their backgrounds.

Step 3: Map the AI image on 3D face mesh using faceMeshTrackers

To achieve a more precise AR face effect, we need to map the AI image on 3D face mesh as our faces are not flat. To do this, Download the Face Reference Assets directly using this link, then unzip the file, and you will see faceMeshTrackers.png in the Textures folder.

Open faceMashTrackers.png in Photoshop and drag your AI image onto it.

I use the free transform tool to warp the image and map it on faceMashTrackers.

Next, export the final image as a PNG file.

Step 4: Create an AR effect using template face mask template

Install Meta Spark AR Studio. Select the “Face Mask” template to create your AR effects.

The guideline for this template are straightforward - simply right-click on "Replace Me" and import your AI face mask image.

Once the image has been imported, the face mask effect will be complete!

You can customize it further by adding other textures, and particle effects, or adjusting the materials as desired. In this example, I add an eye color change effect and face retouch material.

Step 5: Test the AR effect on Instagram

Although you can preview your effect in the Spark Studio, it's always better to test it on Instagram. To do this, go to the left menu bar and click on "Test on Device".

Final Step: Publish the AR effect!

Your AR effect is now ready for publishing! Click on the blue "Publish" button located in the bottom right-hand corner, and carefully review all requirements.

It is important to ensure that the file size is kept under 4MB. Before publishing your effect, review the "Asset Summary" to determine if any modifications, such as materials and 3D models, can be made to reduce the file size.