fix(docx): keep every argument of an OMML delimiter and honour sepChr - #2574
Younes Beriane (drakeo338) wants to merge 1 commit into
Conversation
<m:e> may repeat inside <m:d>, but only the last argument was kept and <m:sepChr> was never read. Join all arguments with the separator, which defaults to "|" per the OMML spec, and treat an empty value as none.
|
@microsoft-github-policy-service agree |
XU (kokokoXUY)
left a comment
There was a problem hiding this comment.
Review: OMML delimiter arguments and sepChr, checked against base and head
I reproduced the report in #2573 and reviewed c62c20cf3e11e165cf4454f0ce37f79f18b7a298 against
its base 1f9530a6d35364886a038dc2dcb34256bb0c85b, using the same interpreter for both trees
(CPython 3.13, PYTHONPATH pointed at each worktree's packages/markitdown/src). I built XML
fragments and a minimal WordprocessingML package by hand, converted them through
MarkItDown().convert_stream, ran the new test file red/green, ran the full
packages/markitdown/tests suite on both sides, and diffed the result of merging with the current
tip of main. The diagnosis is right, the change is the right size, and I could not find a
regression. Details and four non-blocking observations below.
Spec alignment
ECMA-376 says of m:sepChr: "This element specifies the character that separates base arguments
<e> in the delimiter object <d>. If this element is omitted, the default <sepChr> is '|'."
Adding "sep": "|" to D_DEFAULT is therefore the documented default, not an arbitrary choice, and
CT_D really does allow maxOccurs="unbounded" on <m:e>, so dropping the extra arguments loses
document content.
One nuance worth recording in the test file, because the issue text phrases it differently: the
issue says that a sepChr element present without m:val means "no separator", whereas this patch
falls back to | in that case, because get_char(None, default) returns the default. That fallback
matches the <val> attribute text ("When it is omitted, the parent uses its assigned default"), and
CT_Char declares @val as use="required", so such a document is schema-invalid anyway. The new
test that covers "no separator" uses m:val="" (explicitly empty), which does concatenate. No
behaviour change is needed for valid documents; a comment on the test would just pin the intent.
What I verified
Unit level, 30 fragments, raw output kept in _r217_md2574_base.txt / _r217_md2574_head.txt
(base -> head):
| input | base | head |
|---|---|---|
two <m:e>, sepChr="," |
\left(y\right) |
\left(x,y\right) |
two <m:e>, braces, no sepChr |
\left\{x>0\right\} |
\left\{x|x>0\right\} |
three <m:e>, sepChr=";" |
\left(c\right) |
\left(a;b;c\right) |
two <m:e>, sepChr m:val="" |
\left(b\right) |
\left(ab\right) |
one <m:e> |
\left(x\right) |
\left(x\right) |
Also checked and unchanged on both sides: the non-delimiter controls (sSup, sSubSup, rad,
acc, bar, m) and dPr written after the arguments. Changed and correct on head: nested delimiters (\left(x|\left(y|z\right)\right)), separators that need
escaping (_, {, &), separators that do not (,, ;, |, space), and omitted begChr /
endChr still yielding \left. / \right.. The "no m:val on sepChr" case returns
\left(a|b\right); see the note above.
End to end through the docx converter (_r217_md2574_e2e.py): I built a minimal
WordprocessingML package by hand (no python-docx in this environment) with eleven equations, both
inline m:oMath and block-level m:oMathPara, and converted it with
MarkItDown(enable_plugins=False).convert_stream(..., stream_info=StreamInfo(extension=".docx")).
On base, nine of the eleven equations lost every argument except the last one; on head all eleven
keep them, including the $$ ... $$ variants. Separately, the repository fixture
packages/markitdown/tests/test_files/equations.docx produces byte-identical markdown on both trees
(sha256 prefix 8c5894022d77fe38, 240 characters), which is the expected no-op for the
single-argument delimiters it contains.
Red/green at test level. test_docx_math_delimiter.py run against the base tree: 4 failed, 1 passed
(test_single_argument_is_unchanged is the one that already held); against the head tree: 5 passed.
Full package suite with the same test set and only PYTHONPATH swapped: base 9 failed, 1003 passed,
5 skipped; head 5 failed, 1007 passed, 5 skipped. The five failures that remain are identical on
both trees and are local environment gaps, not anything this patch touches: the four
test_cu_converter.py / test_docintel_html.py failures raise MissingDependencyException for the
optional [az-doc-intel] extra (_doc_intel_converter.py:166-169), and
test_module_misc.py::test_speech_transcription is only skipped under GITHUB_ACTIONS
(skip_remote, test_module_misc.py:32-34), so it reaches the network locally. The four new tests
are the only difference between the two runs.
Existing math tests: test_docx_math.py, test_docx_math_accents.py, test_docx_math_symbols.py
and test_docx_omml.py give 15 passed on base and 15 passed on head.
Formatting: black --check at 23.7.0 (the version the repository's pre-commit hook pins) leaves
omml.py, latex_dict.py and the new test file unchanged.
Mergeability: mergeable_state is behind, but git merge-tree --write-tree 4cc9fa1 c62c20c
exits 0 (tree 965b0bd8c25d0c334e83251390b281520204ef5e), so the branch still merges cleanly with
current main. The five commits that touched converter_utils/docx or tests since 1f9530a
only add or rearrange fixtures and do not modify omml.py or latex_dict.py.
CI: the two workflow runs recorded on c62c20c are pre-commit (run id 36900618455) and tests
(run id 36900618631), and both show conclusion=action_required because this is a first-time fork
contribution waiting for approval, so nothing has actually run on CI yet. All numbers above come
from a local run; the empty check list is not a red signal.
Observations (non-blocking)
do_dstill indexesc_dict["dPr"], so a delimiter without<m:dPr>raisesKeyError: 'dPr'.
That is the same on base and head, so this patch neither introduces nor worsens it, but since the
child loop is being rewritten here,c_dict.get("dPr")with the existing defaults would remove
that crash too. The same shape exists indo_f(KeyError: 'fPr'), so it could also be a
separate issue rather than growing this one.- The new tests are unit-level only. The shared fixture contains two
<m:d>elements and both have
a single<m:e>, so the docx pipeline path is not covered by a fixture. The reporter verified it
by editing a copy ofequations.docx; a dedicated small fixture would guard the path users
actually hit. Optional. escape_latex(sep)is a no-op for,,;and|, so ordinary separators pass through
unchanged, and_,{,&are escaped as expected. A literal backslash separator would produce
\left(a\b\right), which LaTeX reads as a control sequence; that is a pre-existing property of
escape_latexand not something this patch needs to solve.- A side effect worth one line in the description: base raises
KeyError: 'e'for a<m:d>that
has<m:dPr>but no<m:e>; with the argument list, head returns\left(\right)instead of
crashing.
How this was produced
_r217_md2574_probe.py (unit matrix + fixture structure dump), _r217_md2574_e2e.py (hand-built
docx package + fixture conversion), _r217_suite.py (full suite against both trees, only
PYTHONPATH swapped) and _r217_checks_2574.py (branch metadata, workflow runs, merge-tree) sit
next to the worktrees with their raw output. AI tooling was used to write and run these checks;
every number above is reproducible from those scripts. I did not run pre-commit itself (its
environment is not installed here), only black --check with the version the hook pins.
Bottom line: the fix is minimal, it matches the documented default, it turns four red tests green
without moving anything else, and it leaves existing fixtures byte-identical. I would merge it.
Review marker: omml-delimiter-sepchr-217.
Fixes #2573.
The OMML delimiter handler (
m:d) kept only the last argument of a multi-argument delimiter, sof(x,y)converted tof\left(y\right), and it ignored a customsepChr. The change keeps every argument and joins them withsepChr(defaulting to|, the OMML default). A regression test covers these cases.No PR template in this repo; I ran the new test file on the committed HEAD (5 passed) and did not run the full suite.