(v0.5.0) 【Starry Night Fireworks】 Thank you all, a thousand stars have gathered at the party!

[heshengtao]

Click here to download the comfyui pack containing the LLM party

点击这里，下载包含LLM party的comfyui便携包

✨v0.5.0✨【星夜烟火】【Starry Night Fireworks】

This release includes the following features:

1. Updated a series of conversion nodes: markdown to HTML, svg to image, HTML to image, mermaid to image, markdown to Excel.
2. Compatible with the llama3.2 vision model, supports multi-turn dialogue, visual functions. Model address: meta-llama/Llama-3.2-11B-Vision-Instruct. Example workflow: llama3.2_vision.
3. Adapted GOT-OCR2, supports formatted output results, supports fine text recognition using position boxes and colors. Model address: GOT-OCR2. Example workflow converts a screenshot of a webpage into HTML code and then opens the browser to display this webpage: img2web.
4. The local LLM loader nodes have been significantly adjusted, so you no longer need to choose the model type yourself. The llava loader node and GGUF loader node have been re-added. The model type on the local LLM model chain node has been changed to LLM, VLM-GGUF, and LLM-GGUF, corresponding to directly loading LLM models, loading VLM models, and loading GGUF format LLM models. VLM models and GGUF format LLM models are now supported again. Local calls can now be compatible with more models! Example workflows: LLM_local, llava, GGUF
5. Added EasyOCR node for recognizing text and positions in images. It can generate corresponding masks and return a JSON string for LLM to view. There are standard and premium versions available for everyone to choose from!
6. In the comfyui LLM party, the strawberry system of the chatgpt-o1 series model was reproduced, referring to the prompts of Llamaberry. Example workflow: Strawberry system compared to o1.
7. A new GPT-sovits node has been added, allowing you to call the GPT-sovits model to convert text into speech based on your reference audio. You can also fill in the path of your fine-tuned model (if not filled, the base model will be used for inference) to get any desired voice. To use it, you need to download the GPT-sovits project and the corresponding base model locally, then start the API service with runtime\python.exe api_v2.py in the GPT-sovits project folder. Additionally, the chatTTS node has been moved to comfyui LLM mafia. The reason is that chatTTS has many dependencies, and its license on PyPi is CC BY-NC 4.0, which is a non-commercial license. Even though the chatTTS GitHub project is under the AGPL license, we moved the chatTTS node to comfyui LLM mafia to avoid unnecessary trouble. We hope everyone understands!
8. Now supports OpenAI’s latest model, the o1 series!
9. Added a local file control tool that allows the LLM to control files in your specified folder, such as reading, writing, appending, deleting, renaming, moving, and copying files.Due to the potential danger of this node, it is included in comfyui LLM mafia.
10. New SQL tools allow LLM to query SQL databases.
11. Updated the multilingual version of the README. Workflow for translating the README document: translate_readme
12. Updated 4 iterator nodes (text iterator, picture iterator, excel iterator, json iterator). The iterator modes are: sequential, random, and infinite. The order will be output in sequence until the index limit is exceeded, the process will be automatically aborted, and the index value will be reset to 0. Random will choose a random index output, and infinite will loop output.
13. Added Gemini API loader node, now compatible with Gemini official API!Since Gemini generates an error with a return code of 500 if the returned parameter contains Chinese characters during the tool call, some tool nodes are unavailable.example workflow:start_with_gemini
14. Added lore book node, you can insert your background settings when talking to LLM, example workflow: lorebook
15. Added FLUX prompt word generator mask node, which can generate Hearthstone cards, Game King cards, posters, comics and other styles of prompt words, which can make the FLUX model straight out. Reference workflow: FLUX prompt word
16. A local file reading tool has been added. In comparison to the previous local file control tool in ComfyUI LLM Mafia, this tool can only read files or the file tree within a specific folder, thus ensuring greater security.
17. Forked chatgpt-on-wechat, created a new repository party-on-wechat. The installation and usage methods are the same as the original project, no configuration is required, just start the party's FastAPI. By default, it calls the wx_api workflow and supports image output. It will be updated gradually to ensure a smooth experience of party on WeChat.
18. Added an In-Context-LoRA mask node, used for generating consistent In-Context-LoRA prompts.
19. We have added a frontend component with features laid out from left to right as follows:
    • Saves your API key and Base URL to the config.ini file. When you use the fix node function on the API LLM loader node, it will automatically read the updated API key and Base URL from the config.ini file.
    • Starts a FastAPI service that can be used to call your ComfyUI workflow. If you run it directly, you get an OpenAI interface at http://127.0.0.1:8817/v1/. You need to connect the start and end of your workflow to the 'Start Workflow' and 'End Workflow', then save in API format to the workflow_api folder. Then, in any frontend that can call the OpenAI interface, input model name=<your workflow name without the .json extension>, Base URL=http://127.0.0.1:8817/v1/, and the API key can be filled with any value.
    • Starts a Streamlit application; the workflow saving process is as above. You can select your saved workflow in the 'Settings' of the Streamlit app and interact with your workflow agent in the 'Chat'.
    • 'About Us', which introduces this project.
20. The automatic model name list node has been removed and replaced with a simple API LLM loader node, which automatically retrieves your model name list from the configuration in your config.ini file. You just need to select a name to load the model. Additionally, the simple LLM loader, simple LLM-GGUF loader, simple VLM loader, simple VLM-GGUF loader, and simple LLM lora loader nodes have been updated. They all automatically read the model paths from the model folder within the party folder, making it easier for everyone to load various local models.
21. LLMs can now dynamically load lora like SD and FLUX. You can chain multiple loras to load more loras on the same LLM. Example workflow: start_with_LLM_LORA.
22. Added the searxng tool, which can aggregate searches across the entire web. Perplexica also relies on this aggregation search tool, so you can set up a Perplexica at your party. You can deploy the searxng/searxng public image in Docker, then start it using docker run -d -p 8080:8080 searxng/searxng, and access it using http://localhost:8080. You can fill in this URL http://localhost:8080 in the party's searxng tool, and then you can use searxng as a tool for LLM.
23. Major Update!!! Now you can encapsulate any ComfyUI workflow into an LLM tool node. You can have your LLM control multiple ComfyUI workflows simultaneously. When you want it to complete some tasks, it can choose the appropriate ComfyUI workflow based on your prompt, complete your task, and return the result to you. Example workflow: comfyui_workflows_tool. The specific steps are as follows:
    • First, connect the text input interface of the workflow you want to encapsulate as a tool to the "user_prompt" output of the "Start Workflow" node. This is where the prompt passed in when the LLM calls the tool.
    • Connect the positions where you want to output text and images to the corresponding input positions of the "End Workflow" node.
    • Save this workflow as an API (you need to enable developer mode in the settings to see this button).
    • Save this workflow to the workflow_api folder of this project.
    • Restart ComfyUI and create a simple LLM workflow, such as: start_with_LLM_api.
    • Add a "Workflow Tool" node to this LLM node and connect it to the tool input of the LLM node.
    • In the "Workflow Tool" node, write the name of the workflow file you want to call in the first input box, for example: draw.json. You can write multiple workflow file names. In the second input box, write the function of each workflow so that the LLM understands how to use these workflows.
    • Run it to see the LLM call your encapsulated workflow and return the result to you. If the return is an image, connect the "Preview Image" node to the image output of the LLM node to view the generated image. Note! This method calls a new ComfyUI on your 8190 port, please do not occupy this port. A new terminal will be opened on Windows and Mac systems, please do not close it. The Linux system uses the screen process to achieve this, when you do not need to use it, close this screen process, otherwise, it will always occupy your port.

本次发行包含如下功能：

1. 更新了一系列转换节点，markdown转HTML，svg转图片，HTML转图片，mermaid转图片，markdown转Excel。
2. 兼容了llama3.2 vision模型，支持多轮对话，视觉功能,模型地址：meta-llama/Llama-3.2-11B-Vision-Instruct，示例工作流：llama3.2_vision
3. 适配了GOT-OCR2，支持格式化输出结果，支持使用位置box和颜色来精细识别文字，模型地址：GOT-OCR2,示例工作流将一张网页的截图转换成了HTML代码再打开浏览器显示这个网页：img2web
4. 大幅调整了本地LLM加载器节点，不用自己选择model type了。重新添加了llava加载器节点和GGUF加载器节点。本地LLM模型链节点上的model type已改成LLM、VLM-GGUF、LLM-GGUF三个选项，对应了直接加载LLM模型、加载VLM模型和加载GGUF格式的LLM模型。重新支持了VLM模型和GGUF格式的LLM模型。现在本地调用可以兼容更多的模型了！示例工作流：LLM_local，llava，GGUF
5. 已加入easyOCR节点，用于识别图中文字和位置，可以生成对应的mask，可以返回一个json字符串供LLM查看，有普通版本和高级版本供大家选择！
6. 在comfyui LLM party中复现了chatgpt-o1系列模型的草莓系统，参考了Llamaberry的提示词，示例工作流：Strawberry system compared to o1。
7. 新增了GPT-sovits节点，可以调用GPT-sovits模型，将文字根据你的参考音频转换为语音，也可以将你微调后的模型路径填入（如果不填就是基底模型进行推理），获得你想要的任意音色。使用时，需要下载GPT-sovits项目和对应的基底模型到本地，然后在GPT-sovits项目文件夹下用runtime\python.exe api_v2.py启动API服务。此外，chatTTS节点被移动到了comfyui LLM mafia中。原因是chatTTS的依赖库较多，且在PyPi中的许可证为CC BY-NC 4.0，这是一个非商用许可证。即使chatTTS的github项目是AGPL协议的，我们还是为了避免不必要的麻烦，将chatTTS节点移到了comfyui LLM mafia中。希望大家能够理解！
8. 现已支持openai最新模型o1系列模型!!!!
9. 新增了本地文件控制工具，让LLM可以控制你指定的文件夹的文件，例如：读取、写入、追加、删除、重命名、移动、复制文件等。由于该节点比较危险，收录在comfyui LLM mafia中。
10. 新增SQL工具，可以让LLM查询SQL数据库。
11. 更新了readme的多语言版本，翻译readme文档的工作流：translate_readme
12. 更新了4个迭代器节点（文字迭代器、图片迭代器、表格迭代器、json迭代器），迭代器模式有：顺序、随机和无限三种模式。顺序会按顺序依次输出，直到超出索引上限自动中止进程，并将索引值重置为0，随机会选择一个随机索引输出，无限会无限循环输出。
13. 新增了Gemini API加载器节点，现在兼容Gemini官方的API啦！如果你是国内网络环境，如果出现API地区受限制的问题，请将节点切换到美国，并使用TUN模式。由于Gemini在工具调用时，如果返回的参数中包含中文字符会出现返回码为500的报错，因此部分工具节点不可用。示例工作流：start_with_gemini
14. 新增lore book节点，可以在与LLM对话时插入你的背景设定，示例工作流：lorebook
15. 新增了FLUX提示词生成器面具节点，可以生成炉石卡牌、游戏王卡牌、海报、漫画等多种风格的提示词，可以让FLUX模型直出。参考工作流：FLUX提示词
16. 添加了本地文件读取工具，相比于之前在comfyui LLM mafia中的本地文件控制工具，这个工具只能读取文件或某个文件夹下的文件树，所以安全很多。
17. fork了chatgpt-on-wechat，新建了一个party-on-wechat，安装和使用方法与原项目一致，无需配置，只需要启动party的fastapi。默认是调用wx_api工作流，支持图片输出。后续会逐步更新，保证party在微信上的丝滑使用。
18. 添加了In-Context-LoRA面具节点，用于生成连续一致性的In-Context-LoRA提示词。
19. 添加了一个前端组件，从左到右的功能分别为：
    • 将你的API key和Base URL保存到config.ini文件，当你对API LLM加载器节点使用fix node后，他会自动读取config.ini文件中你修改后的API key和Base URL。
    • 启动一个fastapi，可以用来调用你的comfyui工作流，如果你直接运行它，你就获得了一个http://127.0.0.1:8817/v1/上的openai接口。你需要将你的工作流的开始和结尾连上开始工作流和结束工作流，然后以API格式保存到workflow_api文件夹，然后在其他可以调用openai接口的前端输入model name=<你的工作流名不包含.json后缀名>，Base URL=http://127.0.0.1:8817/v1/，API key随便填。
    • 启动一个streamlit应用，工作流保存流程如上，你可以在streamlit应用中的设置中选中你保存的工作流，并且在聊天中与你的工作流智能体聊天。
    • 关于我们，介绍本项目。
20. 移除了自动获取模型名称列表节点，取而代之的是简易API LLM加载器节点，自动从你的config.ini文件中的配置中获取你的模型名称列表，你只要选择一个名称就可以加载模型了。此外更新了简易LLM加载器、简易LLM-GGUF加载器、简易VLM加载器、简易VLM-GGUF加载器、简易LLM lora加载器节点。都是自动读取party文件夹的model文件夹内的模型路径，让大家更方便的加载各种本地模型。
21. 现在LLM可以像SD和FLUX一样动态加载lora了，你可以用多个lora串联，以在同一个LLM上加载多的lora。示例工作流：start_with_LLM_LORA。
22. 添加了searxng工具，可以聚合搜索全网，Perplexica也是依赖于这个聚合搜索工具，相当于你可以在party里搭建一个Perplexica。你可以在docker中部署searxng/searxng公共镜像，然后使用docker run -d -p 8080:8080 searxng/searxng来启动它，然后使用http://localhost:8080来访问它。你可以将http://localhost:8080这个URL填入party的searxng工具，就可以将searxng当作LLM的一个工具使用了。
23. 超级重大更新！！！ 现在你可以将任意的comfyui工作流封装成一个LLM工具节点。你可以让你的LLM同时控制多个comfyui工作流，当你想要他完成一些任务时，他可以根据你的提示，选择合适的comfyui工作流，完成你的任务，并且将结果返回给你。示例工作流：comfyui_workflows_tool。具体步骤如下：
    • 首先，将你待会需要被封装成工具的工作流的需要输入文字的接口连上“开始工作流”节点的“user_prompt”输出，这个位置是LLM调用工具时，传入的prompt。在你要输出文字和图片的位置连上“结束工作流”节点的对应输入位置。
    • 将这个工作流以API形式保存（需要再设置中开启开发者模式才能看到这个按钮）。
    • 将这个工作流保存到本项目的workflow_api文件夹中。
    • 重启comfyui，建立一个简单的LLM工作流，例如：start_with_LLM_api。
    • 给这个LLM节点添加一个“工作流工具”节点，连在LLM节点的tool输入上。
    • 在“工作流工具”节点上，在第一个输入框中写入想要调用的工作流文件名，例如：draw.json，可以写入多个工作流文件名。在第二个输入框中写入每个工作流的作用，这样可以让LLM明白这些工作流怎么用。
    • 运行即可看到这个LLM调用你封装好的工作流，并且将结果返回给你。如果返回的是图片，请在LLM节点的image输出上连上“预览图片”节点，查看生成的图片。注意！该方法是在你的8190端口调用了一个新的comfyui，请不要占用这个端口。window和mac系统会有一个新的终端被打开，请不要关闭。Linux系统使用的是screen进程实现的，当你不需要用时，关闭这个screen进程，否则会一直占用你的端口。

---

This discussion was created from the release (v0.5.0) 【Starry Night Fireworks】 Thank you all, a thousand stars have gathered at the party!.