beautifulsoup4

[twkevinzhang]

beautifulsoup4 lib: https://pypi.org/project/beautifulsoup4
The encoding detection capability of lxml lib is not as good as that of bs4, and the official documentation from lxml admits this:

A very nice feature of BeautifulSoup is its excellent support for encoding detection which can provide better results for real-world HTML pages that do not (correctly) declare their encoding.

in Android Pixel 7 device, I got this error. The bad thing is that it's not always reproducible, but it happens quite often lxml.etree.XMLSyntaxError: encoding not supported USC4 little endian, line 1, column 1:

Traceback (most recent call last):
          File "/data/user/0/tw.kevinzhang.news_hub/files/flet/python_site_packages/grpc/_server.py", line 610, in _call_behavior
            response_or_iterator = behavior(argument, context)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/data/user/0/tw.kevinzhang.news_hub/files/extensions/twkevinzhang_komica/api_server_impl.py", line 102, in GetThreadInfos
            raise e
          File "/data/user/0/tw.kevinzhang.news_hub/files/extensions/twkevinzhang_komica/api_server_impl.py", line 96, in GetThreadInfos
            items, _, total = parse_thread_infos_html(result.html, site_id, board_id)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/data/user/0/tw.kevinzhang.news_hub/files/extensions/twkevinzhang_komica/parse_threads.py", line 46, in parse_thread_infos_html
            tree = resolve_usc4_tree(html_content)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/data/user/0/tw.kevinzhang.news_hub/files/extensions/twkevinzhang_komica/parse_threads.py", line 22, in resolve_usc4_tree
            return html.document_fromstring(html_content, parser=parser)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/data/user/0/tw.kevinzhang.news_hub/files/flet/python_site_packages/lxml/html/__init__.py", line 736, in document_fromstring
            value = etree.fromstring(html, parser, **kw)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "src/lxml/etree.pyx", line 3306, in lxml.etree.fromstring
          File "src/lxml/parser.pxi", line 1995, in lxml.etree._parseMemoryDocument
          File "src/lxml/parser.pxi", line 1875, in lxml.etree._parseDoc
          File "src/lxml/parser.pxi", line 1105, in lxml.etree._BaseParser._parseUnicodeDoc
          File "src/lxml/parser.pxi", line 633, in lxml.etree._ParserContext._handleParseResultDoc
          File "src/lxml/parser.pxi", line 743, in lxml.etree._handleParseResult
          File "src/lxml/parser.pxi", line 672, in lxml.etree._raiseParseError
          File "<string>", line 1
        lxml.etree.XMLSyntaxError: encoding not supported USC4 little endian, line 1, column 1

I am sure that bs4 has an irreplaceable position and we need it.

[twkevinzhang]

sorry, beautifulsoup4 look like is a pure python package.