Multiple Cameras Rendering to Bitmaps
Apr 03 (8 months ago)
Ken
wrote
I have a need to (1) Connect to multiple rtsp cameras (2) Render/output each frame of each camera to a bitmap (3) None to all camera outputs may be visible at any given time.
I have looked reasonably closely at the MediaBlocks samples. However, in each example it looks like everything uses/mandates VideoView for rendering. I know I can get the bitmap if I render to the VideoView.
In theory I could just hide windows with a VideoView for each camera if they aren't currently displayed. However, this seems like a waste of time and resources. Therefore I would prefer not to use the VideoView.
It does seem like I should be able to render directly to a bitmap. If so I can't seem to figure out how. Any hints would be appreciated.
I have looked reasonably closely at the MediaBlocks samples. However, in each example it looks like everything uses/mandates VideoView for rendering. I know I can get the bitmap if I render to the VideoView.
In theory I could just hide windows with a VideoView for each camera if they aren't currently displayed. However, this seems like a waste of time and resources. Therefore I would prefer not to use the VideoView.
It does seem like I should be able to render directly to a bitmap. If so I can't seem to figure out how. Any hints would be appreciated.
Reply
15 Answers
Apr 03 (8 months ago)
Roman Miniailov
agent
wrote
Hi Ken,
You can use VideoSampleGrabberBlock to get frames as RGB(A)/Bitmap. VideoView is required to render on screen.
So, remove VideoView and VideoRendererBlock.
Add VideoSampleGrabberBlock like
private VideoSampleGrabberBlock _videoSampleGrabber;
Configure it:
_videoSampleGrabber = new VideoSampleGrabberBlock(VideoFormatX.RGB, addNullRenderer: true);
_videoSampleGrabber.OnVideoFrameBitmap += _videoSampleGrabber_OnVideoFrameBitmap;
_videoSampleGrabber.OnVideoFrameBuffer += _videoSampleGrabber_OnVideoFrameBuffer;
You can use OnVideoFrameBitmap to get Bitmaps or OnVideoFrameBuffer RGB/RGBA/BGRA/etc image data, that can be used to create SKBitmap or some other bitmap type.
Connect the block input instead of the video renderer like:
_pipeline.Connect(someBlock.Output, _videoSampleGrabber.Input);
addNullRenderer: true in the constructor will add a null renderer to close the pipeline. But you can also set it to false and connect something to sample grabber output, for example video encoder.
You can use VideoSampleGrabberBlock to get frames as RGB(A)/Bitmap. VideoView is required to render on screen.
So, remove VideoView and VideoRendererBlock.
Add VideoSampleGrabberBlock like
private VideoSampleGrabberBlock _videoSampleGrabber;
Configure it:
_videoSampleGrabber = new VideoSampleGrabberBlock(VideoFormatX.RGB, addNullRenderer: true);
_videoSampleGrabber.OnVideoFrameBitmap += _videoSampleGrabber_OnVideoFrameBitmap;
_videoSampleGrabber.OnVideoFrameBuffer += _videoSampleGrabber_OnVideoFrameBuffer;
You can use OnVideoFrameBitmap to get Bitmaps or OnVideoFrameBuffer RGB/RGBA/BGRA/etc image data, that can be used to create SKBitmap or some other bitmap type.
Connect the block input instead of the video renderer like:
_pipeline.Connect(someBlock.Output, _videoSampleGrabber.Input);
addNullRenderer: true in the constructor will add a null renderer to close the pipeline. But you can also set it to false and connect something to sample grabber output, for example video encoder.
Apr 03 (8 months ago)
Ken
wrote
Thanks! That was very helpful.
Apr 05 (8 months ago)
Ken
wrote
As long as I'm using only one camera (CPU) memory usage appears to be stable. I am disposing of the original bitmaps I receive from the camera almost immediately.
However, if I use 2+ cameras to receive bitmaps then process memory used VERY quickly expands to unreasonable levels. I can quickly (10 minutes +-) reach process memory of 20Gb or more. I can see it expanding second by second in TaskManager. This never stops until all available memory in consumed.
The C# heap memory levels are stable. CPU usage is obviously somewhat up, but it is nothing unreasonable.
It is possible that this is my code, but I don't think so in that using one camera is completely stable over a period of hours. It is possible that there is some resource contention on my end, but I'm not seeing it.
However, if I use 2+ cameras to receive bitmaps then process memory used VERY quickly expands to unreasonable levels. I can quickly (10 minutes +-) reach process memory of 20Gb or more. I can see it expanding second by second in TaskManager. This never stops until all available memory in consumed.
The C# heap memory levels are stable. CPU usage is obviously somewhat up, but it is nothing unreasonable.
It is possible that this is my code, but I don't think so in that using one camera is completely stable over a period of hours. It is possible that there is some resource contention on my end, but I'm not seeing it.
Apr 05 (8 months ago)
Ken
wrote
One other thing: I am getting frequent exceptions that from the decompile is located in ahy.cs line 223. public override bool Init(PipelineContext a, Pipeline b)
...
if (string.IsNullOrEmpty(this.e.CustomVideoDecoder) && info.VideoStreams.Count > 0)
{
in this case info is NULL. At the very least the function should probably return false instead of throwing an exception. I've tried encapsulating callers in try-catch but have been unable to isolate it. I'm still working on it
...
if (string.IsNullOrEmpty(this.e.CustomVideoDecoder) && info.VideoStreams.Count > 0)
{
in this case info is NULL. At the very least the function should probably return false instead of throwing an exception. I've tried encapsulating callers in try-catch but have been unable to isolate it. I'm still working on it
Apr 05 (8 months ago)
Roman Miniailov
agent
wrote
Please check the v15.9.1 update (update NuGet redists too). The exception happens if you create the RTSP source but set the readInfo flag to false (true by default). If you have no problems, I suggest reading the info during object creation.
But I'm not sure that this exception is related to the memory leak. Please provide a small sample app with code that can reproduce a problem.
But I'm not sure that this exception is related to the memory leak. Please provide a small sample app with code that can reproduce a problem.
Apr 06 (8 months ago)
Ken
wrote
I'm trying 15.9.1 now.
I did try (15.8) reading the info following the RTSP creation. That seemed to do nothing, but maybe I did something wrong. I'll try changing the flag keep having problems.
One other problem (yes, I know) I'm having is that bitmap colors seem very far off. For instance, a bright blue tarp is gold. Green moss is blue-green. This is the same for all cameras. If I open the camera(s) in a browser colors are correct.
I was able to fix the memory problem by introducing delays (lowering the display frame rate). Maybe the GC was just overloaded.
I did try (15.8) reading the info following the RTSP creation. That seemed to do nothing, but maybe I did something wrong. I'll try changing the flag keep having problems.
One other problem (yes, I know) I'm having is that bitmap colors seem very far off. For instance, a bright blue tarp is gold. Green moss is blue-green. This is the same for all cameras. If I open the camera(s) in a browser colors are correct.
I was able to fix the memory problem by introducing delays (lowering the display frame rate). Maybe the GC was just overloaded.
Apr 06 (8 months ago)
Roman Miniailov
agent
wrote
Do you have color issues in our samples with a video preview on screen?
Apr 11 (8 months ago)
Ken
wrote
The same stream displayed to a VideoView draws with the correct colors. The colors from the bitmap are off, but not radically off (with some exceptions). Of necessity I do copy the bitmap immediately after obtaining it, but I do copy to the same format just in case. I don't know what additional information I can provide.
If you have any examples that are using the OnVideoFrameBitmap of the sample grabber for display purposes I'm not seeing that. I did create a very simple example that works with the exception of the bitmap colors.
With 15.9.1 I am still getting the same errors frequently (25% of runs) that I mention with respect to the null reference of "info". I'm not seeing any "readinfo" flag in the RTSPSettings or elsewhere. I may not be looking in the correct spot.
If you have any examples that are using the OnVideoFrameBitmap of the sample grabber for display purposes I'm not seeing that. I did create a very simple example that works with the exception of the bitmap colors.
With 15.9.1 I am still getting the same errors frequently (25% of runs) that I mention with respect to the null reference of "info". I'm not seeing any "readinfo" flag in the RTSPSettings or elsewhere. I may not be looking in the correct spot.
Apr 11 (8 months ago)
Ken
wrote
I did try 15.9.2 as well with that I'm getting exceptions regarding the entry point st_vp9_picture_get_user_data of gstngvcodec.dll not being found. I have also gotten this error on earlier versions when attempting to use a release build.
I wouldn't be shocked if this error has to do with VS not updating something in the solution when the packages are updated. I did try cleaning the solution and a solution re-build. That didn't help. I'll try deleting all the x64 packages and re-installing the packages when I have time.
I wouldn't be shocked if this error has to do with VS not updating something in the solution when the packages are updated. I did try cleaning the solution and a solution re-build. That didn't help. I'll try deleting all the x64 packages and re-installing the packages when I have time.
Apr 12 (8 months ago)
Roman Miniailov
agent
wrote
Hi
Please check the attached sample. I've added VideoSampleGrabberBlock and used it to draw on WPF Image.
You can have incorrect colors because you use RGB in your code, but in the sample grabber, you have BGR or something like it (RGBA/BGRA/etc.).
RTSPSourceSettings.CreateAsync has a "readInfo" optional flag.
Do you have a VP9 stream in your RTSP source? Or using the VP9 NVENC output?
Please check the attached sample. I've added VideoSampleGrabberBlock and used it to draw on WPF Image.
You can have incorrect colors because you use RGB in your code, but in the sample grabber, you have BGR or something like it (RGBA/BGRA/etc.).
RTSPSourceSettings.CreateAsync has a "readInfo" optional flag.
Do you have a VP9 stream in your RTSP source? Or using the VP9 NVENC output?
Apr 12 (8 months ago)
Ken
wrote
OK, the format you are using in the sample is Bgra32. I am using winforms and there is no definition for a bgra PixelFormat in System.Drawing.Imaging.PixelFormat.
In any event, I'm using:
Bitmap copy = new Bitmap(_width, _height, System.Drawing.Imaging.PixelFormat.Format32bppArgb);
using Graphics g = Graphics.FromImage(copy);
g.DrawImage(bitmap, 0, 0, _width, _height);
My understanding is that this should convert the format "automagically". Presumably Winforms just isn't able to handle the format you are outputting. I guess I could use skia to convert the bitmap? I'm not that familiar with it, but I can look into it.
None of my cameras are using VP9. They are all just plain old H264, most highprofile.
In any event, I'm using:
Bitmap copy = new Bitmap(_width, _height, System.Drawing.Imaging.PixelFormat.Format32bppArgb);
using Graphics g = Graphics.FromImage(copy);
g.DrawImage(bitmap, 0, 0, _width, _height);
My understanding is that this should convert the format "automagically". Presumably Winforms just isn't able to handle the format you are outputting. I guess I could use skia to convert the bitmap? I'm not that familiar with it, but I can look into it.
None of my cameras are using VP9. They are all just plain old H264, most highprofile.
Apr 12 (8 months ago)
Ken
wrote
Oops! I forgot to say that I did look at the bitmap provided by the videosamplegrabber before I posted the color related problems. According to WinForms the bitmap was in a rgba32 format. That may have been a misinterpretation by the debugger or System.Drawing.Imaging. However, you might want to double check the PixelFormat flags for the passed bitmap correctly identify it as bgra.
I did manually convert the bitmap to rgba (with some mostly correct help from chat.openai.com). Now the colors are correct.
If you have the option you might want to consider providing an rgba bitmap. They are more widely compatible. If it is too much conversion and/or you don't want to break existing code that's fine. However, you might want to note the format in the documentation.
I did manually convert the bitmap to rgba (with some mostly correct help from chat.openai.com). Now the colors are correct.
If you have the option you might want to consider providing an rgba bitmap. They are more widely compatible. If it is too much conversion and/or you don't want to break existing code that's fine. However, you might want to note the format in the documentation.
Apr 12 (8 months ago)
Roman Miniailov
agent
wrote
Hi
You can change the sample code and use RGBA format in sample grabber settings.
SkiaSharp uses BGRA.
And for buffer data or System Drawing Bitmap you can set RGB also.
You can change the sample code and use RGBA format in sample grabber settings.
SkiaSharp uses BGRA.
And for buffer data or System Drawing Bitmap you can set RGB also.
Apr 13 (8 months ago)
Ken
wrote
I DID select VideoFormatX.RGB in the frame grabber. That is not what I got. I got bgra. The same with RGBA. Again, the bitmap SAID it was RGB, but it was not. I think that this is what is confusing WinForms. It certainly confused me.
Apr 14 (8 months ago)
Roman Miniailov
agent
wrote
Yes, all correct. Thank you!
The problem with Bitmap has been resolved in v15.9.4.
The problem with Bitmap has been resolved in v15.9.4.