fix(docx): Improve text parsing #1268

rateixei · 2025-03-31T09:45:33Z

This PR addresses the issues brought up in #1257, #1275 and #1284:

The text parser now returns the original text when no equations are found (avoiding recreating text that didn't need to be recreated).
Avoids adding NoneType into text array.
When equations are found, instead of recreating the new text (text elements + equations elements), it now inserts the equations in the original text string. This preserves the white spaces in the original text.
Removed warning in boolean/None comparison.
Now also tries to detect headers by checking the style Name in addition to the style ID.
Removes trailing spaces when building in-line group.

Tests are not modified because these are edge cases due to specific white-space formatting.

Checklist:

Documentation has been updated, if necessary.
Examples have been added, if necessary.
Tests have been added, if necessary.

mergify · 2025-03-31T09:46:07Z

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

cau-git

LGTM

ceberam

LGTM

Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com>

#1201) fix wrong type text extracted by tesseract_ocr_cli_model Signed-off-by: gvl4 <Guilhem.VERMOREL@3ds.com> Co-authored-by: gvl4 <Guilhem.VERMOREL@3ds.com> Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com>

Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com>

* chore: bump version to 2.28.4 [skip ci] Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * Improve text parsing Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * fix: Tesseract OCR CLI can't process images composed with numbers only (#1201) fix wrong type text extracted by tesseract_ocr_cli_model Signed-off-by: gvl4 <Guilhem.VERMOREL@3ds.com> Co-authored-by: gvl4 <Guilhem.VERMOREL@3ds.com> Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * Flexibilize heading detection Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * Fix trailing space Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * Remove trailing space Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> --------- Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> Signed-off-by: gvl4 <Guilhem.VERMOREL@3ds.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Guilhem VERMOREL <83694424+guilhemvermorel@users.noreply.github.com> Co-authored-by: gvl4 <Guilhem.VERMOREL@3ds.com> Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com>

…ded to text (#1295) * Adding new latex symbols, simplifying how equations are added to text Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * Identify headers through inhenrited style Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * Log warning message instead of print Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * Adding new latex symbols, simplifying how equations are added to text Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * Identify headers through inhenrited style Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * Log warning message instead of print Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * fix: Tesseract OCR CLI can't process images composed with numbers only (#1201) fix wrong type text extracted by tesseract_ocr_cli_model Signed-off-by: gvl4 <Guilhem.VERMOREL@3ds.com> Co-authored-by: gvl4 <Guilhem.VERMOREL@3ds.com> Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * fix(docx): Improve text parsing (#1268) * chore: bump version to 2.28.4 [skip ci] Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * Improve text parsing Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * fix: Tesseract OCR CLI can't process images composed with numbers only (#1201) fix wrong type text extracted by tesseract_ocr_cli_model Signed-off-by: gvl4 <Guilhem.VERMOREL@3ds.com> Co-authored-by: gvl4 <Guilhem.VERMOREL@3ds.com> Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * Flexibilize heading detection Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * Fix trailing space Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * Remove trailing space Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> --------- Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> Signed-off-by: gvl4 <Guilhem.VERMOREL@3ds.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Guilhem VERMOREL <83694424+guilhemvermorel@users.noreply.github.com> Co-authored-by: gvl4 <Guilhem.VERMOREL@3ds.com> Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * docs: add visual grounding example (#1270) Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * feat(docx): add text formatting and hyperlink support (#630) * feat: Enable markdown text formatting for docx Signed-off-by: SimJeg <sjegou@nvidia.com> * Fix imports Signed-off-by: SimJeg <sjegou@nvidia.com> * Use Formatting Signed-off-by: SimJeg <sjegou@nvidia.com> * Handle hyperlink Signed-off-by: SimJeg <sjegou@nvidia.com> * Handle formatting properly for DocItemLabel.PARAGRAPH Signed-off-by: SimJeg <sjegou@nvidia.com> * Use inline group Signed-off-by: SimJeg <sjegou@nvidia.com> * Handle bullet lists Signed-off-by: SimJeg <sjegou@nvidia.com> * Strip elements Signed-off-by: SimJeg <sjegou@nvidia.com> * Strip elements Signed-off-by: SimJeg <sjegou@nvidia.com> * Run black and mypy Signed-off-by: SimJeg <sjegou@nvidia.com> * Handle header and footer Signed-off-by: SimJeg <sjegou@nvidia.com> * Use inline_fmt everywhere Signed-off-by: SimJeg <sjegou@nvidia.com> * Run precommit Signed-off-by: SimJeg <sjegou@nvidia.com> * Address feedback Signed-off-by: SimJeg <sjegou@nvidia.com> * Fix add_list_item Signed-off-by: SimJeg <sjegou@nvidia.com> * fix minor bugs, mark helper methods internal Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> --------- Signed-off-by: SimJeg <sjegou@nvidia.com> Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> Co-authored-by: Panos Vagenas <pva@zurich.ibm.com> Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * fix(pptx): check if picture shape has an image attached (#1316) Check if picture shape has an image attached in pptx backend Signed-off-by: Maksym Lysak <mly@zurich.ibm.com> Co-authored-by: Maksym Lysak <mly@zurich.ibm.com> Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * chore: update lock file (#1315) chore: update lock Signed-off-by: Michele Dolfi <dol@zurich.ibm.com> Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * docs: add plugins docs (#1319) add plugin docs Signed-off-by: Michele Dolfi <dol@zurich.ibm.com> Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * feat: handle <code> tags as code blocks (#1320) handle <code> tags as code blocks Signed-off-by: FernandoSSI <fernandosi2005@gmail.com> Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * Adding new latex symbols, simplifying how equations are added to text Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * Identify headers through inhenrited style Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * Log warning message instead of print Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> * Adding new latex symbols, simplifying how equations are added to text Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> --------- Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com> Signed-off-by: gvl4 <Guilhem.VERMOREL@3ds.com> Signed-off-by: Panos Vagenas <pva@zurich.ibm.com> Signed-off-by: SimJeg <sjegou@nvidia.com> Signed-off-by: Maksym Lysak <mly@zurich.ibm.com> Signed-off-by: Michele Dolfi <dol@zurich.ibm.com> Signed-off-by: FernandoSSI <fernandosi2005@gmail.com> Co-authored-by: Guilhem VERMOREL <83694424+guilhemvermorel@users.noreply.github.com> Co-authored-by: gvl4 <Guilhem.VERMOREL@3ds.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com> Co-authored-by: Simon Jégou <SimJeg@users.noreply.github.com> Co-authored-by: Panos Vagenas <pva@zurich.ibm.com> Co-authored-by: Maxim Lysak <101627549+maxmnemonic@users.noreply.github.com> Co-authored-by: Maksym Lysak <mly@zurich.ibm.com> Co-authored-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com> Co-authored-by: Fernando Santos <121275806+FernandoSSI@users.noreply.github.com>

rateixei force-pushed the rtdl/improve_text_parsing branch from 14a97bc to 62ade9e Compare March 31, 2025 09:48

rateixei changed the title ~~[WIP] fix(docx): Improve text parsing~~ fix(docx): Improve text parsing Mar 31, 2025

rateixei mentioned this pull request Mar 31, 2025

Error when Converting from DOCX to HTML #1257

Closed

rateixei force-pushed the rtdl/improve_text_parsing branch from 443dd25 to 7a531b4 Compare April 2, 2025 08:35

rateixei marked this pull request as ready for review April 2, 2025 08:37

rateixei requested review from cau-git and ceberam April 2, 2025 08:37

cau-git previously approved these changes Apr 2, 2025

View reviewed changes

rateixei mentioned this pull request Apr 2, 2025

Docx to Markdown, The rusult lost Header's level information left only pure text. #1275

Closed

rateixei dismissed cau-git’s stale review via 613302d April 2, 2025 09:29

rateixei force-pushed the rtdl/improve_text_parsing branch from 613302d to d5c2ce0 Compare April 2, 2025 09:30

rateixei mentioned this pull request Apr 2, 2025

IndexError in MsWordDocumentBackend when processing text without equations #1284

Closed

ceberam previously approved these changes Apr 2, 2025

View reviewed changes

rateixei dismissed ceberam’s stale review via a128b16 April 2, 2025 10:03

github-actions bot and others added 6 commits April 2, 2025 12:04

chore: bump version to 2.28.4 [skip ci]

eb4d17b

Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com>

Improve text parsing

76982a5

Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com>

Flexibilize heading detection

e535209

Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com>

Fix trailing space

331c6ab

Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com>

Remove trailing space

c979eaa

Signed-off-by: Rafael Teixeira de Lima <Rafael.td.lima@gmail.com>

rateixei force-pushed the rtdl/improve_text_parsing branch from a128b16 to c979eaa Compare April 2, 2025 10:04

Merge branch 'main' into rtdl/improve_text_parsing

c0f769c

ceberam approved these changes Apr 2, 2025

View reviewed changes

rateixei merged commit d2d6874 into main Apr 2, 2025
10 checks passed

rateixei deleted the rtdl/improve_text_parsing branch April 2, 2025 10:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(docx): Improve text parsing #1268

fix(docx): Improve text parsing #1268

Uh oh!

rateixei commented Mar 31, 2025 •

edited

Loading

Uh oh!

mergify bot commented Mar 31, 2025 •

edited

Loading

Uh oh!

cau-git left a comment

Uh oh!

ceberam left a comment

Uh oh!

Uh oh!

Uh oh!

fix(docx): Improve text parsing #1268

fix(docx): Improve text parsing #1268

Uh oh!

Conversation

rateixei commented Mar 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify bot commented Mar 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Protections

🟢 Enforce conventional commit

Uh oh!

cau-git left a comment

Choose a reason for hiding this comment

Uh oh!

ceberam left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rateixei commented Mar 31, 2025 •

edited

Loading

mergify bot commented Mar 31, 2025 •

edited

Loading