feat(MiscDrivers,PeriphDrivers,Examples): CSI2/ImgCapture Image Processing Pipeline Rework, New MAX78002 QSPI Example, New RetinaNet Example #583

Jake-Carter · 2023-05-09T23:42:45Z

Overview

This is a big one...

Add APS6404 (external SRAM on the MAX78002EVKIT) drivers.
- This enables a higher capacity buffer destination for captured images.
- The drivers include a refactored SPI implementation ("fastspi") local to the ImgCapture and CSI2 projects.
- aps6404 + fastspi drivers will be moved to MiscDrivers pending @sihyung-maxim's work on the PeriphDrivers SPI API. For now, they will live local to the projects to avoid conflicts.
Refactor CSI2 RevA drivers and MIPI camera API
- These were previously not usable for real-world captures due to poor implementation and significant hardware issues related to the built-in debayering engine.
- Hardware issues have been tracked in Jira, while software workarounds have been put in place to get the drivers functional.
- Abstracts away the complexity of the CSI2 hardware/drivers.
- Decoupled CSI2 drivers from application-side memory management. Application code provides a line handler to offload data to any arbitrary destination.
- Use APS6404 external SRAM as buffer destination for ImgCapture and CSI2 examples
- Improved OV5640 performance by leveraging its built-in debayering engine
Add the PascalVOC-RetinaNetv7_3 object detection demo for the MAX78002
- Captures 320x256 image over CSI2
- Runs RetinaNet object detection
- Draws captured image and overlays bounding boxes to NewHaven TFT display
Add QSPI example for MAX78002EVKIT
Miscellaneous ImgCapture/README fixes

Related PRs

Model training & synthesis files for PascalVOC-RetinaNet

Tests

MAX78002 Internal EVKIT Rev A + Pcam 5C
MAX78002 External EVKIT Rev A + Pcam 5C

RGB565 works at the improved framerate up to 320x320. Beyond that, it works up to 672x514 at reduced framerate.

160x120 (QQVGA) RGB565 (10-15 fps)
320x240 (QVGA) RGB565 (10-15 fps)
320x320 RGB565 (10-15 fps)
640x480 RGB565 (2-3 fps)

RAW8 also works at all the resolutions above. It can push farther up to 800x600 (SVGA) before operation becomes unreliable.

800x600 RAW8 (2-3 fps)
928x768 RAW8 (2-3 fps)

- Working standard reads/writes - Working QSPI reads/writes

- If the value of the dummy byte was defined locally inside of the spi_transmit function, it would introduce some glitches on the SPI MOSI line. Suspected compiler optimization issue. For some reason, these glitches also corrupted the MISO line. This caused RX data to be corrupted, and ram reads would return incorrect values.

…es are corrupted by misaligned IO1)

…n 86us, VGA read in 65us (!!!)

- Working standard reads/writes - Working QSPI reads/writes

- If the value of the dummy byte was defined locally inside of the spi_transmit function, it would introduce some glitches on the SPI MOSI line. Suspected compiler optimization issue. For some reason, these glitches also corrupted the MISO line. This caused RX data to be corrupted, and ram reads would return incorrect values.

…es are corrupted by misaligned IO1)

…n 86us, VGA read in 65us (!!!)

This reverts commit be3c740.

Jake-Carter · 2023-05-18T19:56:46Z

/clang-format-run

Jake-Carter · 2023-05-19T20:50:17Z

Libraries/PeriphDrivers/Source/CSI2/csi2_reva.c


 // Used for Non-DMA CSI2 Cases
 static volatile uint32_t bits_per_pixel;
 static volatile uint32_t fifo_burst_size;
 static volatile mxc_csi2_reva_fifo_trig_t fifo_int_trig;
 static volatile mxc_csi2_ahbwait_t ahbwait_en;

+typedef enum { SELECT_A, SELECT_B } lb_sel_t;


TODO: Delete this (unused)

Libraries/MiscDrivers/Camera/mipi_camera.h

Libraries/MiscDrivers/Camera/ov5640.c

Jacob-Scheiffler · 2023-05-19T21:25:37Z

Libraries/PeriphDrivers/Source/CSI2/csi2_ai87.c

+
+mxc_csi2_capture_stats_t MXC_CSI2_GetCaptureStats()
+{
+    mxc_csi2_reva_capture_stats_t stats = MXC_CSI2_RevA_DMA_GetCaptureStats();


Can you just return stats here? Or return the RevA function call? I forget how the casting works for structs.

mxc_csi2_capture_stats_t MXC_CSI2_GetCaptureStats() { mxc_csi2_reva_capture_stats_t stats = MXC_CSI2_RevA_DMA_GetCaptureStats(); return (mxc_csi2_capture_stats_t)stats; }

Gives

/home/jakecarter/repos/fork/msdk/Libraries/CMSIS/../PeriphDrivers/Source/CSI2/csi2_ai87.c: In function 'MXC_CSI2_GetCaptureStats': /home/jakecarter/repos/fork/msdk/Libraries/CMSIS/../PeriphDrivers/Source/CSI2/csi2_ai87.c:325:5: error: conversion to non-scalar type requested 325 | return (mxc_csi2_capture_stats_t)stats; | ^~~~~~

I think we need to return a pointer to do the type cast but I just wanted to return the struct directly.

I'll see if I can clean this up a bit

Jacob-Scheiffler · 2023-05-19T21:38:52Z

Libraries/PeriphDrivers/Source/CSI2/csi2_reva.c

@@ -67,32 +73,82 @@
    (MXC_F_CSI2_REVA_RX_EINT_VFF_IE_FNEMP_MD | MXC_F_CSI2_REVA_RX_EINT_VFF_IE_FTHD_MD | \
     MXC_F_CSI2_REVA_RX_EINT_VFF_IE_FFUL_MD)

+// A macro to convert a DMA channel number to an IRQn number


Not necessarily for this PR, but we should add this to all <part_number>.h's

Jacob-Scheiffler · 2023-05-19T21:45:17Z

Libraries/PeriphDrivers/Include/MAX78002/csi2.h

- * @brief      Interrupt Handler Function.
- * @return     #E_NO_ERROR if everything is successful.
- */
-int MXC_CSI2_Handler(void);


Is the mxc_csi2_line_handler_cb_t supposed to replace this function?

No, but I wanted to intentionally remove the publicly exposed handler. The drivers are completely interrupt driven from MXC_CSI2_RevAHandler so there is no option to override the CSI2 IRQ handler (yet).

Jake-Carter added 30 commits February 21, 2023 13:22

QSPI initial checkpoint:

ab5f0fc

- Working standard reads/writes - Working QSPI reads/writes

Checkpoint: QSPI+DMA working

6a12c3e

Clear rx buffer between tests

3ff6a4c

Add DMA read (not working - timing issues w/ DMA overhead)

0a93cbf

Checkpoint: register-level DMA write

b23c169

Checkpoint: Working 4-wire reads/writes

8fac207

Add benchmark for DMA overhead (checkpoint: 24us)

9649f4e

Checkpoint: Working QSPI writes, semi-working QSPI reads (first 3 byt…

9ab798d

…es are corrupted by misaligned IO1)

Checkpoint: Debugging QSPI fast read timing

f6d0925

Checkpoint: Working QSPI DMA reads with IPO at 24Mhz SPI, VGA write i…

a7d95b3

…n 86us, VGA read in 65us (!!!)

Break out drivers into their own files

20a8fa4

Add boundary cross test

c1239d3

Move ram reset/exit qspi to sram init

201ee45

Add TODOs

8f2ebc7

Move ram reset/exit qspi to sram init

c9afa64

Add TODOs

f472682

QSPI initial checkpoint:

c9257e8

- Working standard reads/writes - Working QSPI reads/writes

Checkpoint: QSPI+DMA working

c641c88

Clear rx buffer between tests

79626af

Add DMA read (not working - timing issues w/ DMA overhead)

0ee092d

Checkpoint: register-level DMA write

114b151

Checkpoint: Working 4-wire reads/writes

29e2b75

Add benchmark for DMA overhead (checkpoint: 24us)

274f144

Checkpoint: Working QSPI writes, semi-working QSPI reads (first 3 byt…

a619c72

…es are corrupted by misaligned IO1)

Checkpoint: Debugging QSPI fast read timing

5eb7b65

Checkpoint: Working QSPI DMA reads with IPO at 24Mhz SPI, VGA write i…

877d121

…n 86us, VGA read in 65us (!!!)

Break out drivers into their own files

89b48c2

Add boundary cross test

e742c8a

Jake-Carter and others added 5 commits May 15, 2023 17:18

Used signed int8s for CNN input

2b20fe2

Streamline TFT utils args, show initialization indicator on startup

6a2d918

Re-add prints

e46db33

Merge branch 'main' of github.com:Analog-Devices-MSDK/msdk into dev/qspi

cd15613

Revert "clang-format bot reformatting."

54a8168

This reverts commit be3c740.

Jake-Carter and others added 4 commits May 18, 2023 20:00

clang-format bot reformatting.

ac51ef8

Merge branch 'main' of github.com:Analog-Devices-MSDK/msdk into dev/qspi

6788028

Merge branch 'dev/qspi' of github.com:Jake-Carter/msdk into dev/qspi

346d91c

Fix linter errors

60cc43b

Jake-Carter requested review from Jacob-Scheiffler and sihyung-maxim May 19, 2023 14:59

Jake-Carter commented May 19, 2023

View reviewed changes

Jacob-Scheiffler suggested changes May 19, 2023

View reviewed changes

Jake-Carter and others added 5 commits May 22, 2023 14:40

Remove duplicate function headers

8483ec2

Use compiler definition for verification check

121d0f2

ImgCapture: Fix UART transmission bug (Write vs WriteBytes)

62a4b8e

Merge branch 'main' into dev/qspi

a6f2e99

Delete old typedef

04f2bba

Jacob-Scheiffler self-requested a review June 6, 2023 21:25

Jacob-Scheiffler approved these changes Jun 6, 2023

View reviewed changes

sihyung-maxim approved these changes Jun 6, 2023

View reviewed changes

Jake-Carter changed the title ~~CSI2/ImgCapture: Image Processing Pipeline Rework, MAX78002: New QSPI Example, RetinaNet Example~~ feat(MiscDrivers,PeriphDrivers,Examples): CSI2/ImgCapture Image Processing Pipeline Rework, New MAX78002 QSPI Example, New RetinaNet Example Jun 6, 2023

Jake-Carter and others added 4 commits June 6, 2023 17:22

Merge branch 'main' of github.com:Analog-Devices-MSDK/msdk into dev/qspi

7ae8417

Use pull_request_target trigger for labeler workflow

14564f4

Update PR Title Checker Regex (Enforce capital only)

6933776

Add pull_request_target types to labeler workflow

7bd9aad

Jake-Carter merged commit 7c6004b into analogdevicesinc:main Jun 6, 2023

Jake-Carter mentioned this pull request Jun 9, 2023

imgCapture Stream "Image received failed, Only received 3/247808" #614

Closed

Jake-Carter linked an issue Jun 9, 2023 that may be closed by this pull request

imgCapture Stream "Image received failed, Only received 3/247808" #614

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(MiscDrivers,PeriphDrivers,Examples): CSI2/ImgCapture Image Processing Pipeline Rework, New MAX78002 QSPI Example, New RetinaNet Example #583

feat(MiscDrivers,PeriphDrivers,Examples): CSI2/ImgCapture Image Processing Pipeline Rework, New MAX78002 QSPI Example, New RetinaNet Example #583

Jake-Carter commented May 9, 2023 •

edited

Loading

Jake-Carter commented May 18, 2023

Jake-Carter May 19, 2023

Jacob-Scheiffler May 19, 2023

Jake-Carter May 22, 2023

Jacob-Scheiffler May 19, 2023

Jacob-Scheiffler May 19, 2023

Jake-Carter May 22, 2023

feat(MiscDrivers,PeriphDrivers,Examples): CSI2/ImgCapture Image Processing Pipeline Rework, New MAX78002 QSPI Example, New RetinaNet Example #583

feat(MiscDrivers,PeriphDrivers,Examples): CSI2/ImgCapture Image Processing Pipeline Rework, New MAX78002 QSPI Example, New RetinaNet Example #583

Conversation

Jake-Carter commented May 9, 2023 • edited Loading

Overview

Related PRs

Tests

Jake-Carter commented May 18, 2023

Jake-Carter May 19, 2023

Choose a reason for hiding this comment

Jacob-Scheiffler May 19, 2023

Choose a reason for hiding this comment

Jake-Carter May 22, 2023

Choose a reason for hiding this comment

Jacob-Scheiffler May 19, 2023

Choose a reason for hiding this comment

Jacob-Scheiffler May 19, 2023

Choose a reason for hiding this comment

Jake-Carter May 22, 2023

Choose a reason for hiding this comment

Jake-Carter commented May 9, 2023 •

edited

Loading