Hide inputs, include outputs in Markdown format

[mwouts]

The Markdown format for notebooks is a great fit for READMEs on GitHub. For this usage, I would like to be able to include (selected) outputs in the Markdown file. And from time to time I would like to hide a few input cells.

Possible implementation...

• Outputs are delimited with HTML comments. Outputs are ignored when the notebook is loaded, and updated when the notebook is saved.
• Images are created in a configurable directory (say, figure by default?), and the base name for the image is the cell name, if any.
• Code cells are hidden in a HTML comment when marked as not visible by the hide input extension.

[choldgraf]

Just a note that I bet some folks in

https://discourse.jupyter.org/t/generating-reports-for-jupyter-notebooks/279/15

would find this interesting and useful. I was also thinking about improving the story for "one notebook file -> one md file" creation. It seems like one tricky thing about turning certain code blocks / outputs / etc ON or OFF is that this makes two-way sync more difficult.

Do you think there'd be value in building in some functions in jupytext that break two-way synchronization? Otherwise as you mention, you could try keeping track of all the "hidden" pieces by using comments.

The other question is how things like interactive outputs would work in markdown. I guess those would be embedded as HTML snippets?

[mwouts]

Hi, I will soon start thinking of how to best implement this. Currently some ideas I have are listed below, please feel free to discuss those!

Global behavior

• including the outputs in the notebook has to be an explicit choice from the user. We don't want to surprise anyone with this.
• the user should be allowed to retain only certain types of outputs, either per cell or globally for the notebook. For instance, maybe he/she only wants to store the text output. This is something that is already available in nbconvert with the option ``NbConvertBase.display_data_priority(only the first output type that matches the list is retained). Should we use the same option name, i.e.display_data_priority`? Should we allow an user to display the multiple representation for a given output (e.g. text and HTML)?
• Reloading the .md document in Jupyter, and saving again (i.e. a round trip in Jupytext) should preserve the outputs that were stored in the .md document. Similarly, outputs that are not stored in the .md documents should be preserved in the .ipynb file.

Saving outputs

• textual output may be included in the text document itself - provided that it is not too long?
• images, html and javascript content should go to files in a sub folder (provided that they still work fine when we compile the resulting .md file to, say, HTML). If the current document is named sample notebook.md, then following the example of R Markdown we could store the images, etc in a sub-folder sample notebook_files.
• naming the files is very important. For cells that have a metadata name (has to be unique in the notebook) or title, we could use that name as a prefix. Then add -1.png for the first image, -2.png for the second one, etc, as R Markdown does.

Hiding code
I'd like to offer an option to hide one or all code cells. The code cell would obviously remain in the .md document, but in a part of the document that is commented. We should use a standard cell metadata for triggering this, e.g. the tag hide_input as in Jupyter Book.

[mattharrison]

Would love the ability to show output in markdown export (and run doctest on it). I'm authoring a book and this would make be wonderful! :)

[mwouts]

Hello everyone! I have done some research on this subjects and looked into how it would be possible to include (selected or all) outputs in the Markdown file.

I'd follow the default nbconvert's display_data_priority (probably ['html', 'application/pdf', 'svg', 'latex', 'png', 'jpg', 'jpeg', 'text']), and include just one representation of the output, and e.g. drop the text output on plots which looks like <Figure size 432x288 with 1 Axes>, unless the output is required for a visual identity of the notebook (e.g. the JSON data attached to plotly graphs may have to be preserved).

For each output type I plan to offer both inline and include. Include would be the default for images (as this works well in VSCode, PyCharm and GitHub), and inline would be the default for text. For HTML I would prefer to go for include by default, but that won't work on GitHub, so please advice! For Javascript and JSON files I'll choose include by default, as neither include nor inline will work on GitHub (I think, but still have to check, that both forms work in VScode when you turn security settings off).

Now let me describe what I have in mind for each output type

Text

Outputs like

{
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }

could be represented as

```output_execute_result
2
```

similarly to jupyter nbconvert which sets class="output_text output_subarea output_execute_result" on these outputs.

{
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using print\n",
      "using sys.stdout.write"
     ]
    }

could be represented as

```output_stdout
using print
using sys.stdout.write
```

unless we want to preserve the 'name' and 'output_type' more explicitly. Note that jupyter nbconvert sets class="output_subarea output_stream output_stdout output_text" on these outputs.

Similarly, for outputs on stderr we could use pseudo code blocks starting with ```output_stderr. Again, nbconvert sets class="output_subarea output_stream output_stderr output_text"

I also plan to allow including the output in a .html file, that would be useful especially for long outputs like logs.

Images

By default, images would use the standard Markdown inclusion like

![](notebook_name_outputs/unnamed_code_cell_5_1.png)

Jupytext would know that this is an image output just by matching the image name: if it is in the output folder (by default {notebook_name}_outputs) and if the image name matches {code_cell_name}_{output_count}.{mime_ext}.

Inline images exist in Markdown, and work in VSCode, but are not displayed by GitHub.

![](data:image/png;base64,iVBORw0KGgoAAAA...)

HTML

By default we could include the HTML inline, like here:

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	0
0	4

Maybe I should filter out the style section which does not display nicely on GitHub.

I will also offer the option to include the HTML file using <object>:

<object type="text/html" data="notebook_name_outputs/unnamed_code_cell_3_1.html"></object>

or <iframe>:

<iframe src="notebook_name_outputs/unnamed_code_cell_3_1.html" seamless frameborder="0"></iframe>

These kind of work in VSCode (they add an unesthetic extra box), but not on GitHub.

Javascript

By default scripts would be included using <script>:

<script src="notebook_outputs/unnamed_code_cell_12_4.js"></script>

This may work in VSCode, but not on GitHub.

Metadata

I've not discussed metadata here, but we'll need to find a way to store the metadata outputs. Probably using JSON in HTML comments.

Summary

• images are super easy to include, and would work well
• text and html would be supported by GitHub only if we inject them inline, but then we get unreadable diffs again.
• an alternative is to include the text and html outputs (ok with VSCode and pandoc). Which of <object> and <iframe> is the most standard?

[choldgraf]

Is there a way that, instead of creating our own folders etc, we could piggy-back on the ipynb format for including data/images/etc in the text-file representation?

E.g. if we could just treat the ipynb file as a dictionary of unique cell IDs -> outputs. Then we could reference the cell ID in the notebook and its output from within the text file? I'm not sure how that'd work with something like vscode/atom/etc, but if it were possible I think it'd be a nice way to add outputs to the text files without changing the files too much

[mwouts]

Sure, the .ipynb format is a great container. And we already do something like what I think you suggest (if I understood correctly): when the user pairs a notebook to a .md file, we use the .ipynb file for storing the outputs (and filtered metadata), and the .md file for storing inputs. Is that what you were thinking of?

This is available in Jupyter (pair a notebook to a text format, close it and reopen the text document), and also on the command line (jupytext --update). The limitation is that the outputs are associated to the input text itself, so in case the inputs are severly modified (other than code reformating with e.g. black), the corresponding ouputs are removed.