A better way to get letterboxing in Unity

Alternative title: A critique of the existing Unity letterboxing solutions and how to improve them

In a video titled "Handling different aspect ratios in your Unity 2D games", Rabidgremlin pointed out a real issue with aspect ratio scaling in Unity: Different section of your game world is cut off from view when you switch aspect ratio due to Unity trying to preserve the vertical FOV of a camera at every moment. However, the solutions he chose have... problems.

You see, the solution that Rabidgremlin chose for this issue was an asset on the Unity Asset Store called "Auto Letterbox". The way this asset implements letterboxing is: 

- At runtime, in a Start method, find all cameras in a scene using a FindObjectsOfType<Camera> call, save them to a list, modify each of their viewport rect to limit what they can display to fit the target aspect ratio.

- Instantiate a new camera, set its background color to the color you would choose for your letterbox, set its depth to a low value of -100, set its culling mask to 0 to show nothing other than the background with the chosen color.

- Run an Update loop during which the viewport rect of all saved cameras is recalculated and the letterbox color is re-applied.

In a later video titled "Letter Box: Free asset to handle aspect ratios in Unity 2D", Rabidgremlin introduced an asset of his own that applies letterboxing in a very similar methodology. The actual name of this asset is "Letter Boxer", and it differs from Auto Letterbox in the following ways:

- The initial viewport rect of a letterboxed camera is done in an Awake method instead of Start.

- The letterboxing is done on a per-camera basis, the LetterBox component therefore has to be attached to each of the chosen camera. However, FindObjectsOfType<Camera> is still called once for each LetterBox component to find cameras with conflicting -100 depth.

- The author - Rabidgremlin himself - only guarantees this asset to work with orthographic cameras.

As far as I'm aware, these are the 2 only existing letterboxing solutions for Unity.

I have the following critique of both solutions:

- By instantiating a camera, calling FindObjectsOfType and running an Update loop all on runtime, they raise some performance concerns.

- They restrict the use of letterbox areas to solid block of colors. You don't have the choice to use a background image for the letterbox areas.

 

Pictured: Ikaruga. This port of a vertical arcade shoot 'em up not only uses background image for the letterbox area, it displays additional UI in that area.

- They neither help you get tight a control on your scenes, nor help you get a good understanding of "how the sausage is made".

That sums up the motivation for this post - to present a better way to get letterboxing in Unity (title zing!).

First, let's think about this for a moment. The first goal of my alternative letterboxing methodology is to be more minimalist and more "native" to Unity's paradigm. Unity UI has this component called AspectRatioFitter. Hmm... are you thinking what I'm thinking?

We start with a Canvas. Any type of Canvas, using any configuration for its Canvas Scaler. Under it, create an empty GameObject. This will be the aspect ratio-fitted area that we view the world through, so let's name it "Aspect Ratio Fitted Container" (we shall call this the ARFC from now on). Make the ARFC stretch to all edges of the Canvas. To make this object actually fit itself to an aspect ratio, add an AspectRatioFitter to the ARFC, fill in the result of width / height of your target aspect ratio in the "Aspect Ratio" field, and pick "Fit In Parent" for the mode of the AspectRatioFitter. For example: 0.5625 would be the value for the "Aspect Ratio" field if you're making a mobile game and targetting 9:16 aspect ratio.

This is how your setup should look like at this step.

We are going to add the letterboxes next. Add a blank Image under the ARFC, give it a color that you would have given to your letterbox, set its Min Anchor to (0, 1), set its Max Anchor to (1, Iota). Iota for the purpose of this post is a sufficiently-large positive number (Iota is not used as an actual standardized mathematical notation). Set all of its Left, Right, Top, Bottom numbers to 0 (let's call these the offset numbers). Finally, make sure this Image has "Raycast Target" ticked, and then rename it "Top Bar". At this step, you should be able to realize where we're going with this.

Create the Bottom, Left and Right equivalents of the Top Bar. They will all have the offset numbers set to 0, Raycast Target-enabled Image, same color as the Top Bar. The anchors for all 4 bars will be as following:

• Top: Min Anchor (0, 1); Max Anchor (1, Iota)
• Bottom: Min Anchor (0, -Iota); Max Anchor (1, 0)
• Left: Min Anchor (-Iota, 0); Max Anchor (0, 1)
• Right: Min Anchor (1, 0); Max Anchor (Iota, 1)

At this point, you should have a scene like this:

Right, so we're half-way through. There is still the issue of Unity cameras persisting their vertical FOV under all aspect ratios. You can see that issue in the clip above too. But before we get to that, there are some more not-immediately-obvious setups for the letterbox bars that we have to take care of first.

Create a Canvas under the ARFC - it should be stretched out to all edges of the ARFC by default if you add it from the right-click menu but if it isn't, stretch it out to all edges. Make sure this Canvas doesn't have a Canvas Scaler but does have a Graphics Raycaster. Next, tick on "Override Sorting" and give this Canvas an Iota value for its "Sort Order" field. I gave it 999, high number enough to block everything underneath it, but still slightly below the default sort order of Unity In-Game Debug Console which is a tool I love a lot and recommend for every Unity developers. (yes, my opinion may be biased by the fact that I am a contributor to this project, and no, the author of this project did not sponsor me to promote his creation, I just love it)

Put the 4 bars we created above under this sub-canvas. You can rename this sub-canvas as "Letterbox Bars" if you like. This will make it so that the letterbox bars we created will be excluded from Canvas redraw when there are changes made under the rest of the ARFC hierarchy. Finally, to make working with other objects in the scene less annoying and intrusive in the Scene tab, disable the visibility and selectability of the Letterbox Bars. This is how the scene hierarchy should look at this step:

We are done with the letterboxes themselves here. Now let's think about how to make a camera automatically adjust its FOV to keep a consistent view of the aspect ratio-fitted viewport.

We don't want to run an Update loop just to constantly check the screen's current dimensions every frame to do a recalculation, that would be unnecessary performance overhead. We only want to recalculate the FOV once when the screen dimensions change, such as when you enable split-screening on Android. Is there a callback in Unity Engine for that?

Unfortunately, there is not. Well, there is no direct callback for when screen dimensions change, BUT there is something very close to it in the engine. You see, there is this callback called OnRectTransformDimensionsChange. Its description reads:

This callback is called if an associated RectTransform has its dimensions changed.

The call is also made to all child rect transforms, even if the child transform itself doesn't change - as it could have, depending on its anchoring.

Are you thinking what I'm thinking?

What if we listen to this callback on the ARFC?

After all, the RectTransform of the root Canvas will inevitably have its dimensions change whenever the screen dimensions change, and the ARFC is sure to receive this callback as the result.

Well, if you already got the idea and just want the TLDR code, here it is:

// Author: Tony Giang

using UnityEngine;

using UnityEngine.UI;

using MyBox;

/// <summary>

/// This component keeps a target camera view fit for UGUI letterboxing in

/// accordance to the aspect ratio of the fitter. If the new aspect ratio is

/// wider than the fitter's aspect ratio, the target camera's vertical FOV will

/// be kept consistent. If the new aspect ratio is taller than the fitter's

/// aspect ratio, the target camera's vertical FOV will be stretched.

/// </summary>

[ExecuteAlways]

public class AutoLetterboxCameraFOVFitter : MonoBehaviour

{

  [AutoProperty] public AspectRatioFitter Fitter = null;

  public Camera TargetCamera = null;

  public static float CurrentRatio => (float)Screen.width / Screen.height;

  public float ReferenceVerticalFOV;

  void OnRectTransformDimensionsChange()

  {

    if (CurrentRatio < Fitter.aspectRatio)

    {

      var referenceHorizontalFOV = Camera.VerticalToHorizontalFieldOfView(

        ReferenceVerticalFOV,

        Fitter.aspectRatio);

      TargetCamera.fieldOfView = Camera.HorizontalToVerticalFieldOfView(

        referenceHorizontalFOV,

        CurrentRatio);

    }

    else TargetCamera.fieldOfView = ReferenceVerticalFOV;

  }

  [ButtonMethod]

  public void Fill() => ReferenceVerticalFOV = TargetCamera.fieldOfView;

}

Add this component to the ARFC, assign the target camera you want to auto-correct the FOV of and fill in a reference vertical FOV. AutoProperty and ButtonMethod are attributes from the MyBox package, another awesome package, even though my opinion may be biased by the fact that I am a contributor to this project. You can remove these attributes and manually assign the Fitter field if you don't want to use MyBox.

Save the scene once and you should be done at this stage. The game view now looks like this:

I also wrote a version of this script for orthographic cameras. I provide both scripts as a unitypackage file in this link: https://mega.nz/folder/FK5AFQAZ#LlFDHLccoV-2SpaVHcO0iQ

And there you have it - a minimalist letterbox approach that allows you to use background images for the letterbox areas, doesn't have the runtime overhead of instantiating a new object, FindObjectsOfType or an Update loop. AND you walk away actually understanding how the sausage is made.

If you want to know the nitty gritty details about how the above script works, stick around for a breakdown.

The AutoProperty attribute for the Fitter field by default will autofill with the first component of its type down the hierarchy, starting with its own gameobject. Since AutoLetterboxCameraFOVFitter is meant to be added to the ARFC, it will automatically pick up the aspect ratio that you picked for your your game from the AspectRatioFitter also attached to the ARFC.

The ButtonMethod attribute for the Fill method will turn it into an Inspector button to help you quickly copy the target camera's current FOV as the reference vertical FOV.

Within the OnRectTransformDimensionsChange callback: under the if check is the scenario where the top and bottom letterboxes appear, which means this is where we recalculate the target camera's vertical FOV. The recalculation is made with the assumption that when the screen is taller than its target aspect ratio, we will be trying to preserve the horizontal FOV of the target camera, not its vertical FOV. Hence, we calculate the desired horizontal FOV that we would have gotten if the target camera had the reference vertical FOV and the screen had the target aspect ratio. Then we back-convert that horizontal into a vertical FOV that would be needed to get the same horizontal FOV in the current screen aspect ratio.

The else scenario is where the left and right letterboxes appear, we just have to reapply the reference vertical FOV in this case.

The ExecuteAlways attribute is added to the component class to help you preview how other things - especially the rest of the UI - scale with different aspect ratios without launching the game into Play mode. This cuts down on the dev-test cycle.

And finally, before you ask: yes, the OnRectTransformDimensionsChange callback is always called at least once when the canvas is first constructed. That's why there is no need to calculate the target camera's FOV in Start or Awake.

Thank you for reading. This topic will return on this blog. Scaling UI for a letterboxed game is technically not a part of letterboxing itself, but you'll regret not knowing how to do that too.