The max resolution for the H.264 stream depends on two different factors: camera capabilities and client capabilities.
Camera side
The maximum resolution the hardware encoder of the P6 platform can encode is 2048x1536 px (QXGA).
Hence, for single lens cameras, QXGA corresponds also to the maximum resolution that can be selected.
If you configure the camera to produce a live stream with higher resolution than 2048x1536 px the H.264 module will automatically downscale that stream to fit into the QXGA limitation of the encoder.
For example, a stream with 4096x1536 px (2 x QXGA) is simply not possible in H.264.
Therefore if the camera is setup to deliver a dual lens stream and QXGA is selected for the H.264 stream within the Integration Protocol page, the resulting stream will have a resolution of 2048x768 px.
How did we get to that value?
As we said, 2048x1536 px is the maximum resolution.
QXGA = 2048x1536 whereas 2 x QXGA would mean 4096 x 1536.
4094 is x2 the max resolution, therefore the camera will divide this size by 2.
In order to maintain the aspect ration, the camera will divide by 2 the vertical size too: 1536 / 2 = 768
There is nothing we can do for H.264 to overcome this limitation in the P6 platform.
As opposed to H.264, MJPEG does not have such a limitation.
Client side
The max resolutions for the RTSP protocol is FullHD (1920x1080).
To reach higher resolutions (up to 6 MP), the client needs to support the Onvif protocol.