diff options
| author | Moudy Ho | 2023-03-27 11:13:24 +0800 |
|---|---|---|
| committer | Greg Kroah-Hartman | 2023-05-11 23:03:15 +0900 |
| commit | affad9e791019a3560f807e17d9d515ca4f1dd36 (patch) | |
| tree | d64bc5eb30aa53ef5f4eecefcf007062846dcc8f /drivers | |
| parent | 5a72aea9acfe945353fb3a2f141f4e526a5f3684 (diff) | |
media: platform: mtk-mdp3: fix potential frame size overflow in mdp_try_fmt_mplane()
[ Upstream commit 4168720753ce6c14c5d3a35302fc2e1841383443 ]
Fix overflow risk when setting certain formats whose frame size exceeds
a RGB24 with 7723x7723 resolution.
For example, a 7723x7724 RGB24 frame:
1. bpl (byte per line) = 7723 * 3.
2. Overflow occurs when bpl * 7724 * depth.
Fixes: 61890ccaefaf ("media: platform: mtk-mdp3: add MediaTek MDP3 driver")
Signed-off-by: Moudy Ho <moudy.ho@mediatek.com>
Signed-off-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Signed-off-by: Sasha Levin <sashal@kernel.org>
Diffstat (limited to 'drivers')
| -rw-r--r-- | drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c | 10 |
1 files changed, 6 insertions, 4 deletions
diff --git a/drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c b/drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c index 4e84a37ecdfc..36336d169bd9 100644 --- a/drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c +++ b/drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c @@ -4,6 +4,7 @@ * Author: Ping-Hsun Wu <ping-hsun.wu@mediatek.com> */ +#include <linux/math64.h> #include <media/v4l2-common.h> #include <media/videobuf2-v4l2.h> #include <media/videobuf2-dma-contig.h> @@ -428,14 +429,15 @@ const struct mdp_format *mdp_try_fmt_mplane(struct v4l2_format *f, u32 bpl = pix_mp->plane_fmt[i].bytesperline; u32 min_si, max_si; u32 si = pix_mp->plane_fmt[i].sizeimage; + u64 di; bpl = clamp(bpl, min_bpl, max_bpl); pix_mp->plane_fmt[i].bytesperline = bpl; - min_si = (bpl * pix_mp->height * fmt->depth[i]) / - fmt->row_depth[i]; - max_si = (bpl * s.max_height * fmt->depth[i]) / - fmt->row_depth[i]; + di = (u64)bpl * pix_mp->height * fmt->depth[i]; + min_si = (u32)div_u64(di, fmt->row_depth[i]); + di = (u64)bpl * s.max_height * fmt->depth[i]; + max_si = (u32)div_u64(di, fmt->row_depth[i]); si = clamp(si, min_si, max_si); pix_mp->plane_fmt[i].sizeimage = si; |