Baking rigged meshes to static, custom world thumbnails, security improvements

Hello everyone and welcome to another of our weekly updates!

While  we are internally finalizing one of the educational experiences for  public release, there's a number of general features and improvements  that have been sprouted from this work, that should benefit anyone using  Neos.

You can now bake rigged (posed) meshes into a static one!  They keep their current pose and blendshapes when baked, essentially  becoming a mesh snapshot. There's already an easy to use tool made by  Orange that lets you easily strike a pose and generate a figurine of  your avatar. It behaves as any other mesh in Neos, so it can not only be  used for fun, but also to optimize your worlds or export your avatar in  a pose for 3D printing or rendering!

Related to this,  SkinnedMeshRenderer has a new Explicit bounds mode. This can be used to  fix cases where animated meshes would randomly get culled when they  shouldn't, without the performance penalty of dynamically computing its  bounds every frame.

We also added ability to customize the world  thumbnails! This makes it easier to ensure a good thumbnail for any  worlds you publish to help entice players to visit them. Not only it  helps our own official worlds, but any others as well, including the  incoming MMC submissions as it's nearing its end.

Based on  community reports, we also improved security of the initial session  connection when joining users get authenticated and fixed issue where  importing malicious .blend files could run Python scripts in the  embedded file. As a reminder, you can report any security issues at moderation.neos.com according to our Security Reporting Policy.

Explicit Skinned Mesh Renderer bounds

During  our work on the educational demos, we have noticed an issue with a  number of animated rigged meshes - they would de-render at random times  and simply not be visible, even though they were there.

This was  due to them using a static bounding box of the mesh in its default  (binding) pose for culling, which didn’t correspond to where the mesh  actually was positioned by its bones as part of the animation.

One  approach for this is to simply recompute the bounding box dynamically  every frame. However this has a noticeable performance cost, which adds  up with more meshes in the scene.

To deal with this better, we  implemented the capability to perform skinned mesh transformations in  our own data structures, which enables a whole bunch of applications and  features.

Notably it now lets you pre-compute an explicit  bounding box on the current pose of the mesh, to ensure that it’s always  rendered properly without an extra per-frame performance cost. You can  even expand the current bounding box to incorporate a new pose with a  press of the button.

The explicit bounds can be manually edited  with hand too if you know how and where the mesh should be rendered,  though it is not recommended to simply make it cover a huge area unless  absolutely necessary, otherwise the mesh will need to be calculated even  when you cannot see it.

Baking Skinned Meshes into a static pose

Perhaps  a much more fun feature to sprout from this new internal addition is  the ability to bake skinned meshes into a static mesh in its current  pose and with its current blendshapes. It’s something that’s been  requested by a number of community members as well.

For example  this lets you take your avatar and simply bake yourself into a static  mesh in whatever pose and expression you strike at the moment. The  resulting mesh is like any other. You can even give it a collider  afterwards!

Our Japanese community has already built a tool on  top of this feature - simply step on a pedestal, press a button and in a  few seconds a small figurine of your avatar pops out! Check out the  video below if you’d like to see it in action.


https://twitter.com/mikan3134/status/1438804932858023938

Another  practical use case comes from another of the projects that our team has  been involved in - the Metamovie. There are a number of corpses lying  around the world, all of them skinned meshes, fully posed directly in  Neos.

However previously we had to export those out and bake them  outside of Neos and reimport, which is a tedious process, but improved  the world’s performance. Now with this tool baking them is a one-click  operation directly in Neos.

To actually bake a mesh, you can  click a button on the SkinnedMeshRenderer. Alternatively, to bake  multiple meshes (both static and skinned) you can use the Glue tool in  the “Bake Meshes” mode or the LogiX node for baking.

Customizable World & Session Thumbnails

Yet  another long-requested feature has been prodded into implementation  with our educational project - being able to customize the thumbnail of  your saved world and locations of session thumbnails.

Thumbnails  can be important when publishing a world, as they can be the deciding  factor whether someone visits your world or not, but previously they  would always be captured at whatever point you were in the world when  hitting the Save button.

There are now new sets of components  that let you override this. If you’d still like to have the most recent  thumbnail captured whenever you save, you can use the  WorldCaptureThumbnailSource component.

This will capture a fresh  thumbnail from wherever this component is located. You can even add an  optional overlay texture on top of it. Alternatively you can use  WorldTextureThumbnailSource to simply provide a static 360 texture for  the thumbnail if you made a polished one.

We also added  SessionCaptureThumbnailSource. You can scatter these components around  popular locations in your world, so whenever a thumbnail is captured for  the session, it will be taken from one of those points. The system will  pick a random one every time, but prioritize locations with more people  nearby than others.

Improved session connection handshake and other security improvements

Based  on security reports through our moderation ticket system, we  implemented more security improvements. Notably we reworked a portion of  the protocol that establishes the initial connection to a session.

This  process performs a cryptographic authentication of any user joining the  session, to ensure that they are who they say they are. However, part  of this process was susceptible to a man-in-the-middle attack, which  while difficult to execute, would allow someone to impersonate another  user in a different session.

This mechanism has now been expanded  to significantly mitigate the possibility of such attack as well as  number of others, providing greater security guarantees on user  identities.

Additionally we also improved security of importing  .blend files. Blender files can contain Python scripts that run  automatically when they are opened. When the file is imported directly  to Neos, those scripts no longer run automatically, preventing attack  where importing a malicious .blend file could run arbitrary code on your  computer.

If you find any security issues, please report them responsibly at moderation.neos.com!  Reports are eligible for CDFT rewards, depending on combination of  their severity, quality of the report and other factors. You can learn  more about our Security reporting policy on our GitHub here.

Bug Fixes, small optimizations and tweaks

Among  those larger additions and improvements, we released a number of other  bug fixes, tweaks and additions. For example during the work on the  skinned mesh renderer, we noticed unnecessary data being stored on each  bone binding and removed it, reducing some memory usage and GC workload.

We  improved the avatar IK feet behavior, to avoid the legs going up when  near another player, but still keep reacting to other parts of their  body - for example their hands if they’re pushing another player’s feet  up.

We fixed a number of crashes too and  DynamicBonePlayerColliders components were ignored after recent rework  of the system. We updated the Assimp importer library to the latest as  well, bringing about 150 commits of bug fixes and additions that should  improve compatibility with various 3D model formats.

If you’d  like to know more, you can check out our release notes on Steam or  Discord! We hope that everyone had a great week and that the next one  will be awesome too and as always - huge thanks to you for helping to  keep this project going. Neos is a living, breathing metaverse and we're  amazed at the creativity we see every day!