Video Texture Analyses (in performace context), 2024

Hello,

I am working with video and audio, together.

And wanting to use some kind of analyses to extract info about a video (playing as a texture), and use this to control aspects of sound.

There are so many rich tools for doing this in the other direction. When working with sound, there is feature detection of every conceivable precision and contour, and so highly optimized.

I am wondering what approaches people take when they want the Video to influence the Audio.

Which features do you extract? And, how do you go about doing this?

I am specifically wondering about doing this with textures, in a 'realtime' performance patch.

Are there GPU side solutions to get brightness, or color values, or do 'frame differencing', or some kind of 'activity level' in a video playing as a texture?

I am aware of the jit.gl.asyncread -> jit.3m, kind of 'jitter approach'.... as I see used in the Vizzie ANALYZR / vz.analyzr module.

But, is that the best or only way? If you have a fairly big patch running multiple streams of video and audio and processing each with multiple effects, and streams of control data, etc., etc, etc.. I just assume it's not best practice to be reading all frames back to CPU. (My understanding of all this is vague and rudimentary, so please forgive me.)

Any suggestions, or examples, or pointers to packages or tools I may have missed, are very much appreciated.

Thank you!!

🌞