rST directive `.. contents` often generates wrong links (Confluence Cloud)

[flichtenheld]

Given a document like

Testing Sphinx Confluence Builder 2.13.0 (with parentheses)
---------------------------------------------------------------------------------------

.. confluence_toc::

rst TOC:

.. contents:: Table of Contents

Overview
--------

Some content with `internal link`_.
And a second link: `other values`_.

Some Section
----------

Just add some text.

2.59.0 (Ubuntu 22)
++++++++++++++++++

Section title that contains parentheses.

The TOC generated by .. contents: often generates wrong links.

When using confluence_editor = v1, the correct links (according to confluence_toc) are:

#TestingSphinxConfluenceBuilder2.13.0(withparentheses)-Overview
#TestingSphinxConfluenceBuilder2.13.0(withparentheses)-Some-Space
#TestingSphinxConfluenceBuilder2.13.0(withparentheses)-2.59.0(Ubuntu22)

However, the links generated for .. contents:: are:

#Overview
#SomeSpace
#2.59.0(Ubuntu22)

When using confluence_editor = v2, the correct links are:

#%5BinlineExtension%5D%5BinlineExtension%5DOverview
#%5BinlineExtension%5D%5BinlineExtension%5DSome-Space
#2.59.0-(Ubuntu-22)

The links generated for .. contents:: are:

#Overview
#Some-Space
#[inlineExtension][inlineExtension]2.59.0-%28Ubuntu-22%29

(So exactly the wrong way around).

Note that for pages generated originally with confluence_editor = v1 and then updated with confluence_editor = v2, the behavior becomes even more confusing. The "correct" links are then:

#TestingSphinxConfluenceBuilder2.13.0(withparentheses)-OverviewTestingSphinxConfluenceBuilder2.13.0(withparentheses)-OverviewOverview
#TestingSphinxConfluenceBuilder2.13.0(withparentheses)-Some-SpaceTestingSphinxConfluenceBuilder2.13.0(withparentheses)-Some-SpaceSomeSpace
#TestingSphinxConfluenceBuilder2.13.0(withparentheses)-2.59.0(Ubuntu22)

Which might be a bug in Confluence?

[jdknight]

This is all incredibly frustrating. Reference logic has been updated so many times to deal with changes in Confluence and compatibility between v1 and v2 editor variants. A lot of testing was performed to make sure all links to talk to each other no matter what editor variant was used (where we replicated extra anchors in v2 to make everything happen).

And now again... the Cloud (v2) editor seemed to have different identifiers again. When going through a release, validation testing is performed to sanity check references a bunch. If I look at the validation content now, some links are no longer working. The bonus of performing encoding tweaks to handle Confluence v2 "wontfix" bugs and odd inlining markings makes none of this fun.

Probably need to do a rework and major simplification of reference management in this extension. Since the legacy editor will no longer be supported on Cloud, maybe it's time to just drop completely the v1 editor experience for Confluence Cloud targets. Then we no longer have to deal with mixed editor scenarios and reduce the maintenance error on how Confluence evolves.

[flichtenheld]

Are there technical reasons you can't generate manual anchors for sections? Sounds like the only way forward that offers some sanity. Although that probably raises questions about name spaces, to make sure not to conflict with existing ones...

[jdknight]

It should be possible. We kind of do that now for some links (i.e. we generate some extra anchors on a v2 page to support linking from a v1 target to a v2 target based on some requirements of the v1 ac:link anchor usage).

Since this extension supported internal references, it always relies on the identifiers generated by Confluence when possible. The goal was to "properly" use the same target that would be consistent between page generation and UI interaction. I wonder if we should always be generating our own managed anchor identifiers (ideally, normalized value of named entries if possible). I cannot recall what happens if there is an identifier conflict (e.g. a section named "About" that had an ID "about" and having a custom macro with "about"; might be it appends a .1 or is ignored; results varied per editor). The only other trick to deal with is anchor placement. For example, section anchors need to be in the title tag or they will generate newlines on a page.

I'll try to find some time to see if I can get the anchors in a state which I believe were working before. After making a release, reworking/simplifying anchors under the v2 editor and drop support for multi-editor links.

[jdknight]

Changes (#1127) have been made to help always generate anchor links no matter what the configured editor type. This should ensure compatibility anchors are created for all sections and appear to improve reference handling issues observed in Confluence Cloud. This change has been added into the main branch and should be made available next stable release. This can be tested now by using the development version. Feel free to try it out ahead of time any report any issues you may have.

[flichtenheld]

So with v2 this seems to work correctly for "normal" section titles. But as soon as there is a ( in the section title no anchor is generated at all. This is clearly a bug on your side since the code you generate simply does not contain the anchor code. Haven't done any extensive testing with different special characters, but ( definitely triggers it.

Example output:

<h3><ac:structured-macro ac:name="anchor">
<ac:parameter ac:name="">Level3</ac:parameter>
</ac:structured-macro><ac:structured-macro ac:name="anchor">
<ac:parameter ac:name="">TestingSphinxConfluenceBuilder2.14.0dev0v2-Level3</ac:parameter>
</ac:structured-macro><ac:link ac:anchor="id7">
<ac:link-body>Level 3</ac:link-body>
</ac:link></h3>
<p>More nesting.</p>
<h3><ac:link ac:anchor="id8">
<ac:link-body>Level 3 (with special char)</ac:link-body>
</ac:link></h3>

With editor version v1 I get no working links. This looks to me more to be a Confluence-side bug, since now the code you generate for both versions is basically identical. Given that the "official" anchor used by confluence_toc for v1 documents is now something like #TestingSphinxConfluenceBuilder2.14.0dev0v1-Level3TestingSphinxConfluenceBuilder2.14.0dev0v1-Level3Level3 maybe it is just time to give up on v1. Because Atlassian clearly seems to do so...

[jdknight]

Made another adjustment (#1129) where we will no longer use the Confluence-generated anchor target for "special" characters and always use the first section identifier (that we added in the previous changeset). Merged into the development branch if you would like to try.

[flichtenheld]

This new change seems to be only about generating the links but does not seem to address my issue with the dev version that the section anchors are not created for sections with special characters as I mentioned above (#1121 (comment)).

FWIW there is some weirdness going on with the links but without having the anchors I can't say whether it works as intended or is buggy: When I have a section name "Level 3 (with special char)" then the link created is #level-3-with-special-char which seems sane. But when I have two sections named "2.59.0 (Ubuntu 22)" and "2.58.0 (Ubuntu 22)" the links created are #ubuntu-22 und #id1? Why does it ignore the number part?

[jdknight]

For sure, the changes recently added are more so focused on ensuring links are functional with respect to users just clicking links. The extension does not aim to provide a consistent anchor experience if that is one of the things you may be looking for.

section anchors are not created for sections with special characters

This is a limitation of Confluence Cloud as outlined in CONFCLOUD-74698 (closed by Atlassian as WONTFIX; at the time of writing). Anchors with characters like parenthesis will be automatically removed (this does not happen on Confluence Data Center). While encoding characters can help prevent Confluence from auto-removing them, linking to them can be problematic when using ac:link macros.

When I have a section name "Level 3 (with special char)" then the link created is #level-3-with-special-char which seems sane. But when I have two sections named "2.59.0 (Ubuntu 22)" and "2.58.0 (Ubuntu 22)" the links created are #ubuntu-22 und #id1? Why does it ignore the number part?

This extension will try to map to Atlassian-generated anchors. However, the recent change for sections with characters like parenthesis, it will now only rely on the using the unique identifier values generated by Sphinx/docutils. I would speculate that docutils does the following:

• Converts "Level 3 (with special char)" to local-3-with-special-char.
• When generating an ID for "2.59.0 (Ubuntu 22)", it ignores leading numbers which results in an identifier ubuntu-22.
• When generating an ID for "2.58.0 (Ubuntu 22)", it does the same but since ubuntu-22 is taken, it then picks the next free identifier based on id<N>, which in this case is id1.

[jdknight]

This issue has brought up a couple of bugs in the extension and some oddities in Confluence Cloud (the report is appreciated). It is believed that the issues with links when using .. contents:: should be addressed. Since these changes have been added in the implementation and are included in the most recent release, this issue will be marked as closed for this milestone effort.

However, any more comments/feedback can be added here if something is missed (if desired). If there is a another/new bug or new enhancement related to this, maybe we can create a new issue to track the effort.