Multiple Progress instances

[akhileshraju]

I would like to use multiple Progress instances to show different progress meters in the console.
Each Progress instance may/may not have more than 1 task in it.

From the code (and the exception I encountered) I realized that Progress.__init__ creates a new Live instance
https://github.com/willmcgugan/rich/blob/5d716be414bff0aee9db75e409f2443f7c506a5c/rich/progress.py#L598

Before doing any of these things, in my application, I always create an instance of Console and use it for any rich related printing in my code.
The very next step after creating the Console instance, I also create a Live instance and use the console.set_live() method.

Coming back to the multiple Progress instances - While creating new instances, I pass the console I created in the above step to Progress.__init__. However, instead of using the _live from the console, it seems to create a live instance nonetheless which results in a "Only one live instance can be active" exception when I try the second Progress instance and start it.

I am not sure how to proceed and any help in this regard is appreciated 🙂

[willmcgugan]

You can only have a single Live active at any one time. But you can have a renderable which contains multiple progress bars.

Have a look at this example which has two progress bars in a Layout. Although you should be able to do it by having your progress bars in a Table, or RenderGroup.

[willmcgugan]

Better still, look at this example

[ODtian]

Hi, I'm happy to see this question,. I've noticed the approach in the example at very begining and write my code as well. However after catching so many OS error 24 (too many open file), considering I have to create many Progress instance to show progress bar in tree structure and monitoring file handles both in linux and windows I found it is too many Live created in __init__ that cause the issue.

So please allowing me to ask what is the point that Live have to create file handles? Or wheather should we add a arg to send custom live to Progress?

[willmcgugan]

It's unlikely to be an issue in Live or Progress. If you show me code I may be able to give one suggestions.

[ODtian]

from rich.console import Group
from rich.filesize import pick_unit_and_suffix
from rich.progress import (
    BarColumn,
    DownloadColumn,
    Progress,
    ProgressColumn,
    TextColumn,
    TimeRemainingColumn,
    TransferSpeedColumn,
)
from rich.text import Text
from rich.tree import Tree

from ..args import Args
from ..console import log_error, multilive
from ..utils import save_failed
from .base import BaseBar, BaseProgressManager


class LiveProgress(Progress):
    def __init__(
        self,
        *columns,
        console=None,
        auto_refresh=True,
        refresh_per_second=10,
        speed_estimate_period=30.0,
        transient=False,
        redirect_stdout=True,
        redirect_stderr=True,
        get_time=None,
        disable=False,
        expand=False,
        live=None,
    ):
        super().__init__(
            *columns,
            console=console,
            auto_refresh=auto_refresh,
            refresh_per_second=refresh_per_second,
            speed_estimate_period=speed_estimate_period,
            transient=transient,
            redirect_stdout=redirect_stdout,
            redirect_stderr=redirect_stderr,
            get_time=get_time,
            disable=disable,
            expand=expand,
        )
        # without this, file handles will increase as the bar created
        self.live = live or self.live

    def start(self):
        pass

    def stop(self):
        pass


class CustomUnitDownloadColumn(ProgressColumn):
    def __init__(self, unit):
        super().__init__()

        if isinstance(unit, tuple):
            self._suffixes, self._base = unit
        else:
            self._suffixes = [
                u + unit for u in ("", "K", "M", "G", "T", "P", "E", "Z", "Y")
            ]
            self._base = 1000

    def render(self, task):
        completed = int(task.completed)
        total = int(task.total)
        unit, suffix = pick_unit_and_suffix(total, self._suffixes, self._base)

        completed_ratio = completed / unit
        total_ratio = total / unit
        precision = 0 if unit == 1 else 1
        completed_str = f"{completed_ratio:,.{precision}f}"
        total_str = f"{total_ratio:,.{precision}f}"
        download_status = f"{completed_str}/{total_str} {suffix}"
        download_text = Text(download_status, style="progress.download")
        return download_text


class CustomUnitTransferSpeedColumn(ProgressColumn):
    def __init__(self, unit):
        super().__init__()

        if isinstance(unit, tuple):
            self._suffixes, self._base = unit
        else:
            self._suffixes = [
                u + unit for u in ("", "K", "M", "G", "T", "P", "E", "Z", "Y")
            ]
            self._base = 1000

    def render(self, task):
        speed = task.finished_speed or task.speed
        if speed is None:
            return Text("?", style="progress.data.speed")
        unit, suffix = pick_unit_and_suffix(speed, self._suffixes, self._base)
        speed_ratio = speed / unit
        precision = 0 if unit == 1 else 1
        speed_str = f"{speed_ratio:,.{precision}f}"
        return Text(f"{speed_str} {suffix}/s", style="progress.data.speed")


def create_rich_bar(unit=None, live=None):
    return LiveProgress(
        "[progress.percentage]{task.percentage:>3.0f}%",
        BarColumn(),
        CustomUnitDownloadColumn(unit=unit) if unit else DownloadColumn(),
        TextColumn("[yellow] · [/]"),
        CustomUnitTransferSpeedColumn(unit=unit) if unit else TransferSpeedColumn(),
        TextColumn("[yellow] · [/]"),
        TimeRemainingColumn(),
        TextColumn("[yellow] · {task.description}"),
        live=live,
    )


class CommonBar(BaseBar):
    def __init__(self, description, total=0, unit=None, parent_tree=None, remove=True):
        self.description = description
        self.total = total
        self._parent_tree = parent_tree
        self._remove = remove
        self._unit = unit

        self.__bar = None
        self.__tree = None
        self.__task = None

    @property
    def _bar(self):
        if not self.__bar:
            self.__bar = create_rich_bar(unit=self._unit, live=multilive.live)
        return self.__bar

    @property
    def _tree(self):
        if not self.__tree:
            self.__tree = self._parent_tree.add(self._bar)
        return self.__tree

    @property
    def _task(self):
        if not self.__task:
            task_id = self._bar.add_task(self.description, total=self.total)
            self.__task = self._bar.tasks[task_id]
        return self.__task

    def create_sub_bar(self, description, total=0, unit=None):
        return CommonBar(description, total=total, unit=unit, parent_tree=self._tree)

    def reset(self):
        self._bar.reset(self._task.id)

    def complete(self):
        self.update(self._task.remaining if self._task else self.total)

    def update(self, advance=None, total=None, completed=None):
        self._bar.update(
            self._task.id, advance=advance, total=total, completed=completed
        )

        if self._tree and self._remove and self.is_finished():
            self._parent_tree.children.remove(self._tree)

    def is_finished(self):
        return self._task and self._task.finished


class TreeChildren:
    def __init__(self, tree):
        self._tree = tree
        self._children = []

    def append(self, item):
        self._children.append(item)
        self._tree.on_update()

    def remove(self, item):
        self._children.remove(item)
        self._tree.on_update()

    def __iter__(self):
        return iter(self._children)


class RootTree:
    def __init__(self, on_update):
        self.on_update = on_update
        self.children = TreeChildren(self)

    def add(self, item):
        tree = Tree(item)
        self.children.append(tree)
        return tree


class CommonProgressManager(BaseProgressManager):
    def __init__(self):
        self._tree = RootTree(self.update)
        self._live_id = multilive.start_with(Group())
        self.failed = []

    def update(self):
        multilive.update(self._live_id, Group(*self._tree.children))

    def create_file_bar(self, file):
        return CommonBar(file.position, total=file.size, parent_tree=self._tree)

    def create_sub_bar(self, description, total=0, unit=None):
        return CommonBar(
            description, total=total, unit=unit, parent_tree=self._tree, remove=False
        )

    def error(self, e, max_retry_exceeded=False):
        log_error(e)

        if max_retry_exceeded:
            self.failed.append(e.position)

    async def close(self):
        if self.failed:
            save_bar = self.create_sub_bar(
                "Save failed files", total=len(self.failed), unit="Files"
            )
            save_failed(self.failed, Args.FAILED_SAVE_PATH)
            save_bar.complete()

        multilive.stop(self._live_id)

in console.py

class MultiLive:
    def __init__(self, console=None):
        self.live = Live("", console=console)
        self._content = {}
        self._live_id = 0

    @property
    def _group(self):
        return Group(*map(lambda i: i[1], sorted(self._content.items())))

    def _update_live(self):
        self.live.update(self._group)

    def update(self, live_id, item):
        self._content[live_id] = item
        self._update_live()

    def start_with(self, item):
        live_id = self._live_id
        self._live_id += 1

        self.update(live_id, item)

        if not self.live.is_started:
            self.live.start()

        return live_id

    def stop(self, live_id=None):
        if live_id:
            self._content.pop(live_id)
            self._update_live()

        if self.live.is_started:
            self.live.stop()


console = Console()
multilive = MultiLive(console)

[willmcgugan]

That's a bit too much to debug here and its not self contained, but I can see you're doing a few things that may be problematic. I'd suggest comparing your solution to the live_progress example.

[ODtian]

All right, but all live created finally closed according to my debuging, and add an argument to custom live may resolve my problem.
Always, thanks for your work.

[ODtian]

Hi, after all the day working on it I still didn't figure out why there are so many file handles. Finally I convince myself that it's a bug in rich and write a simple test which is really short but reproduce the problem:

import time

from rich.live import Live
from rich.progress import Progress


def progress():
    _ = [Progress() for i in range(1000)]
    time.sleep(15)


def live():
    _ = [Live() for i in range(1000)]
    time.sleep(15)


live()

progress() create 1000 Progress instance and do nothing, but windows task manager showed that there will be exactly 2000 more file handles created. Moreover, live() create exactly 1000 less file handles just a half of Progress one which doesn't look like a coincidence. I've run the test both on linux and windows using asyncio or not for several times and gave stable results. Even I execute del _ to delete all instance the results are still the same.

Screenshots

live()

progress()

do nothing, just sleep.

What on the left side of "已禁用" is the number of file handles.

[boegel]

This question is close to the one I had in #1500, where I eventually figured it out myself, see the example I included there.