How to properly check if a material is used in a scene?

Sometimes you may need to know if a given material is used within the scene or not. In Compact Material Editor it is usually marked with the UI giving you certain clues. For example, in the screen capture below, the gray corners indicate the material is used in the scene. Corners that are filled in white, indicate the material is used, and also is being used by a selected scene object. The white rectangle indicates it is the currently selected material.

The same indicators are found in the Slate Material editor. They will display on the preview of the material.

The obvious question now is how to retrieve this information using the C++ SDK? Let's take a look at how to do it.

The very first step is to have a look at the MtlBase class where you will notice the int TestMtlFlag(int mask) that "Tests the specified flags. Returns nonzero if the flags are set; otherwise zero” . Among the available Materials flags we can notice the MTL_IN_SCENE flag, that seems to be exactly what we need.

Thus, to test if a material is used or not we can easily do:

if(my_material->TestMtlFlag(MTL_IN_SCENE))
{
    \\ It is used
} else
{
    \\it is not
}

Hurrah! Job is done, let's go home.

Not so fast.

The MTL_IN_SCENE flag is useful only if somebody sets this flag (calling MtlBase::SetMtlFlag), but it turns out that nobody sets this specific flag, like ever. Thus, it will always return 0, no matter what, and even if you have assigned this material to dozens of objects in the scene, it is never accurate, unless someone sets it (maybe you set it in your plugin, but is seems none of the default materials are doing this.)

Other (Materials) flags seems to be working fine, thus being useful, but not this one. So, let's get back to work.

This question is not new, and there are lot of (mostly MAXScript) recipes on how to check whenever a given material is used somewhere in the scene, and the common algorithm is to get the scene materials and check if the material in question is found among them.

That should be piece of cake, as the Interface Class offers us the GetSceneMtls method, and the returned MtlBaseLib has the handy FindMaterial method for finding a material by name:

bool materialIsUsed(MtlBase* this_material)
{
   MtlBaseLib* scene_materials = GetCOREInterface()->GetSceneMtls();

   return (scene_materials->FindMtl(this_material) != -1) ? true : false;
}

Easy say, easy do - can we go home now?
Not yet ... not yet ...

It turns out that this approach is not very reliable (noticed by many developers), and the reason is that there is no dynamic tracking of the materials in the scene, thus the material library returned by the Interface::GetSceneMtls might not contain the latest information.

There is a way to manually reset and rebuilding the scene lib, but unfortunately it is available only in MAXScript as ‘UpdateSceneMaterialLib()’method, and there is not yet a C++ SDK equivalent.

"So, what it is the big deal? We can execute MAXScript from C++, we can live with that."

Well ... there is yet another problem.

If the material in question is a part of a sub-material, then you can check in the scene materials lib whenever you want, it will not show up, because (in a manner of speaking) it contains only the “first level” materials and will not contain the materials that are part/contained in other materials (g.e. multi-material).

“Then, what could be the solution?”

The solution is given us by the Reference System, because if we look at MtlBase inheritance diagram:

we notice the obvious aspect, that any material (like many other classes in 3ds Max SDK) is also a Reference Target.

This is obvious, because logically, a material has to have dependents, that are notified upon change (i.e. change the diffuse color).

Thus, what is left is to iterate through these dependents, using the dedicated DependentIterator Class and recursively check the dependents.

Easy say, but not so obviously done:

static bool  isNodeInScene(INode* node)
{
       for (INode* par = node->GetParentNode();
              par != nullptr;
              node = par, par = node->GetParentNode()) {
       }
       return node == GetCOREInterface()->GetRootNode();
}

 static bool isUsedInScene(ReferenceTarget *mt)
{
       DependentIterator di(mt);
       RefMakerHandle rm;

       if (mt->SuperClassID() == BASENODE_CLASS_ID) {
              return isNodeInScene(static_castINode*>(mt));
       }

       while ((rm = di.Next()) != NULL) {
              if (rm != mt && rm->IsRefTarget()
                     && isUsedInScene(static_castReferenceTarget*>(rm)))
                     return true;
       }
       return false;
}

Thus, to check if a given material is used in the scene or not, it suffice to call:

if(isUsedInScene(my_material))
{
    \\ It is used
}else
{
    \\it is not
}

and the cool thing with this approach is that (I guess you already noticed) it can be used with anything that is “referable” (i.e. is a ReferenceTarget), and that could be more interesting than just for materials.