Using tags from Pluto notebook metadata #1010

stefanbringuier · 2023-04-20T22:03:45Z

stefanbringuier
Apr 20, 2023

I have a site that I generate using Franklin.jl which lists links to HTML formatted pluto notebooks stored in assets. Recently, Pluto.jl added the ability to include metadata fields such as tags, which show up in the HTML exported pluto notebooks as:

<meta property="og:article:tag" content='notebook'>

I was wondering how to make it so the generated tag list in Franklin.jl includes these tags? For example, this is what I have so far.

In the utils.jl file I define the following:

using Gumbo
using Cascadia
using Glob

function parse_tags_from_html(filepath::String)
  html_content = read(filepath, String)
  html = parsehtml(html_content)
  tags_selector = Selector("meta[property='og:article:tag']")
  tags_nodes = eachmatch(tags_selector, html.root)
  return map(x -> get(x.attributes, "content", ""), tags_nodes)
end


function hfun_get_tags()
  tags = Set{String}()

  # Retrieve tags from HTML Puto notebooks
  html_files = glob("*.html", "./_assets/notebooks")
  for html_file in html_files
    file_tags = parse_tags_from_html(html_file)
    union!(tags, file_tags)
  end

  # Sort and collect the tags
  sorted_tags = sort(collect(tags))

  # Generate the HTML for the tag list
  html_buffer = IOBuffer()
  for tag in sorted_tags
    write(
      html_buffer,
      """
<li class="tag-item">
  <a class="tag-link" href="/tag/$(tag)/">$(tag)</a>
</li>
"""
    )
  end

  return String(take!(html_buffer))
end


function hfun_custom_taglist()
  # Add this loop to process the HTML Pluto.jl notebooks
  html_files = glob("*.html", "./_assets/notebooks")

  # Grab all tag names first from the notebooks
  tags  = []
  for html_file in html_files
    push!(tags,parse_tags_from_html(html_file)...)
  end
  
  # Process the tags and integrate them into the tag list as needed
  # For example, adding the tags to the existing rpaths list
  rpaths = []
  for html_file in html_files
    for tag_from_html in parse_tags_from_html(html_file)
      if tag_from_html in unique(tags)
        push!(rpaths, html_file)
      end
    end
  end

  c = IOBuffer()
  write(c, "<ul>")
  for rpath in rpaths
    title = let 
      path = split(rpath, '/')[end]
      name, _ = split(path, '.')
      name
     end
    write(c, "<li><a href=\"/$rpath/\">$title.jl</a></li>")
  end
  write(c, "</ul>")
  return String(take!(c))
end

I then use get_tags in header.htmlto show the tags on the home page, this works in that it displays the tags as desired and seems to provide the proper relative path. Then in order to actually create the pages for the tags I have the following in the tag.html:

<!doctype html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="stylesheet" href="/css/franklin.css">
  <link rel="stylesheet" href="/css/basic.css">
  <title>Tag: {{fill fd_tag}}</title>
</head>
<body>
  {{insert header.html}}
  <div class="{{div_content}}">
    <h1>Tag: {{fill fd_tag}}</h1>
    {{custom_taglist}}
    {{insert page_foot.html}}
  </div>
</body>
</html>

The issue is I'm not seeing any tag folder when using serve to inspect. I'm thinking the issue is I none of the global variables for tags are present, i.e., fd_tag of fd_page_tags.

Any help is much appreciated.

Answered by tlienart

May 9, 2023

@stefanbringuier kindly update to get the latest commit on tags-i1010. The attached utils.jl file has a few modifications and should work. A few notes:

rpath is the relative path to the root of the website in unix/url format. For markdown given that a page foo.md will eventually lead to a foo/index.html which will be accessed at foo/, the rpath is foo; for an assets page however this is not the case, and so the rpath contains the extension, therefore my logic of adding *.html was wrong (ended up checking if file.html.html existed)

this is the core of the error that remained.

Modifications in utils.jl:

given the comment above, the sorter(p) = begin ... logic in hfun_taglist_pluto should…

View full answer

tlienart · 2023-04-21T08:23:55Z

tlienart
Apr 21, 2023
Collaborator

Nice, I think there's three things I'd change:

I'd change your logic a bit to a hfun_get_pluto_tags that you call in your index.md which goes over the notebooks, retrieves the tags as you do, and pushes them to fd_page_tags (see code below)

Code to push to fd_page_tags:

Franklin.set_var!(Franklin.GLOBAL_VARS, "fd_page_tags", Franklin.DTAG((rpath => tags,)); check=false)

In Franklin the function generate_tag_pages ( here ) currently has logic to check the keys of fd_page_tags to verify that the source page exists. That logic assumes that the rpath should be extended with .md whereas in your case it's not a .md file.

This requires a change; I've added on a branch tags-i1010 a small change that allows you to put the following in your config.md:

tag_source_exts = [".html"]

You can try that by adding Franklin@tags-i1010

your custom tag.html which has a custom_taglist --> simplify that custom_taglist since the parsing will have already been done; just adjust the logic from hfun_taglist

If it doesn't work, it might be good to point me to a simple repo where I can try playing with the code a bit to hopefully get stuff to work

If it does work, let me know and I can merge that branch into master and do a patch release.

1 reply

stefanbringuier Apr 21, 2023
Author

@tlienart wow thanks for this! Okay it seems to have worked! I used the branch and have the function:

function hfun_get_pluto_tags()
  html_files = get_html_files()
  tags = []
  for html_file in html_files
    rpath = replace(html_file, "_assets" => "assets")
    push!(tags, rpath => Set(parse_tags_from_html(html_file)))
  end
  Franklin.set_var!(Franklin.GLOBAL_VARS, "fd_page_tags", Franklin.DTAG(Tuple(tags)); check=false)
  return nothing
end

It gets called in index.md and I used hfun_taglist structure.

The only peculiarity is that even though notebooks and their tags get added by hfun_get_pluto_tags, it appears generate_tag_pages is ingoring some of them. For example, after calling the function above , GLOBAL_VARS["fd_pages_tags"] shows:

OrderedCollections.LittleDict{String, Set{String}, Vector{String}, Vector{Set{String}}}("./assets/prerendered_notebooks/AtomicCalculationWorkflow.html" => Set(["Simulation", "Atomistics"]),"./assets/prerendered_notebooks/RemakeBestPracticesPost.html" => Set(["Materials data", "Machine Learning", "Materials Informatics", "Materials Science"])) => (OrderedCollections.LittleDict{String, Set{String}, Vector{String}, Vector{Set{String}}},)

but when I got to tag/Atomistic or tag/Simulation they aren't generated pages (i.e. 404). The other tags from the second entry work.

tlienart · 2023-04-22T12:26:34Z

tlienart
Apr 22, 2023
Collaborator

Nice, it looks like the types are a bit different, extracting from your paste:

"./assets/prerendered_notebooks/AtomicCalculationWorkflow.html" =>
     Set(
        ["Simulation", "Atomistics"]
),

but

"./assets/prerendered_notebooks/RemakeBestPracticesPost.html" => 
     Set(
        ["Materials data", "Machine Learning", "Materials Informatics", "Materials Science"]))
        =>
        (OrderedCollections.LittleDict{String, Set{String}, Vector{String}, Vector{Set{String}}},
)

Could you check what your tags vector looks like and make sure it's consistent? maybe the types returned by parse_tags_from_html are inconsistent?

What is used internally is:

tags = Set(refstring.(locvar(:tags)::Vector{String}))
# and later
... DTAG((rpath => tags,)) ...

Franklin.refstring is just something that makes tags like "Machine Learning" into machine_learning (you can test this with Franklin.refstring("Machine Learning")

So what I'd do is check that your parse_tags_from_html returns a Vector{String} (and if not, which I suspect is the case, adjust so that it does); then use Set(refstring...) as above, then try the whole thing again and see if it improves?

4 replies

stefanbringuier Apr 24, 2023
Author

You are right, I see the type difference now, however, I checked and parse_tags_from_html does indeed return tags of Vector{String} for all cases (i.e. notebooks).

The other thing is that the case

"./assets/prerendered_notebooks/RemakeBestPracticesPost.html" => 
     Set(
        ["Materials data", "Machine Learning", "Materials Informatics", "Materials Science"]))
        =>
        (OrderedCollections.LittleDict{String, Set{String}, Vector{String}, Vector{Set{String}}},
)

Is actually the scenario where the tags/pages are working, its the other case that is not working!

Thanks again for the excellent support much appreciated.

tlienart Apr 24, 2023
Collaborator

Did you try using the refstring function?

failing that would you mind pointing me to a branch where I could try playing around to see if I can get this to work?

stefanbringuier Apr 24, 2023
Author

No, not yet will do.

You could play around with this branch.

stefanbringuier Apr 28, 2023
Author

So I tried refstring and restructured the rpath/tags using Franklin.DTAG(zip(rpaths,map(Set,tags))), I get:

OrderedCollections.LittleDict{String, Set{String}, Vector{String}, Vector{Set{String}}}(
    "./assets/prerendered_notebooks/AtomicCalculationWorkflow.html" => Set(["atomistics", "simulation_workflows"]),
    "./assets/prerendered_notebooks/RemakeBestPracticesPost.html" => Set(["machine_learning", "materials_science", "materials_informatics", "materials_data"])
)

Which the type seems correct, which gives GLOBAL_VARS["fd_page_tags"] as:

OrderedCollections.LittleDict{String, Set{String}, Vector{String}, Vector{Set{String}}}( 
   "./assets/prerendered_notebooks/AtomicCalculationWorkflow.html" => Set(["atomistics", "simulation_workflows"]), 
   "./assets/prerendered_notebooks/RemakeBestPracticesPost.html" => Set(["machine_learning", "materials_science", "materials_informatics", "materials_data"])
) => (OrderedCollections.LittleDict{String, Set{String}, Vector{String}, Vector{Set{String}}},)

Which seem correct based on set_var! with the conditions check=false and isglobal=false gives Pair(value,(T, ))

The problem is the tags atomistics and simulation_workflows still don't have pages but the other tags do.

tlienart · 2023-05-05T17:20:45Z

tlienart
May 5, 2023
Collaborator

sorry I couldn't look into this earlier, could you update the branch with your work so I could start from that?

1 reply

stefanbringuier May 8, 2023
Author

No worries at all.

The add-tags branch is updated with my effort to use refstring.

Again, thanks for your time.

tlienart · 2023-05-09T10:48:30Z

tlienart
May 9, 2023
Collaborator

@stefanbringuier kindly update to get the latest commit on tags-i1010. The attached utils.jl file has a few modifications and should work. A few notes:

rpath is the relative path to the root of the website in unix/url format. For markdown given that a page foo.md will eventually lead to a foo/index.html which will be accessed at foo/, the rpath is foo; for an assets page however this is not the case, and so the rpath contains the extension, therefore my logic of adding *.html was wrong (ended up checking if file.html.html existed)

this is the core of the error that remained.

Modifications in utils.jl:

given the comment above, the sorter(p) = begin ... logic in hfun_taglist_pluto should look at rpath directly, not at rpath * .html
the logic around parse_title_from_html should be refactored a bit; your parse_title_from_html returns an empty string when it doesn't find a <title> so I changed the isnothing to a isempty; getting a fallback title was not ideal either your nb_name does something weird, I didn't fix that but you should
in your _layout/tag.html there's the default {{fill fd_tag}} which will now point to the refstring'ed title so something like machine_learning; fixing this is pretty easy, just add something like {{tag_title}} instead of {{fill fd_tag}} and define a hfun_tag_title which gets locvar(fd_tag) and converts it back to what it should be. Instead of inverting the logic of refstring, what you could do is have a dictionary inutils.jl which you populate gradually in taglist_pluto which maps fd_tag => tag_title.

I hope this sets you up to get this working fine! sorry it was a bit of a mess...

Note:

you can remove tag_source_exts = [".html"] from config.md it's not needed anymore.
first call do serve(clear=true) to make sure your __site is rebuilt from scratch

using Gumbo
using Cascadia
using Glob
using Franklin

function hfun_bar(vname)
  val = Meta.parse(vname[1])
  return round(sqrt(val), digits=2)
end

function hfun_m1fill(vname)
  var = vname[1]
  return pagevar("index", var)
end

function lx_baz(com, _)
  # keep this first line
  brace_content = Franklin.content(com.braces[1]) # input string
  # do whatever you want here
  return uppercase(brace_content)
end


get_html_files() = vcat(glob("*.html", "./_assets/notebooks"), glob("*.html", "./_assets/prerendered_notebooks"))

nb_name(rpath) = begin
  file = split(rpath, '/')[end]
  split(file,'.')[1]
end

function parse_tags_from_html(filepath::String)
  html_content = read(filepath, String)
  html = parsehtml(html_content)
  tags_selector = Selector("meta[property='og:article:tag']")
  tags_nodes = eachmatch(tags_selector, html.root)
  return map(x -> get(x.attributes, "content", ""), tags_nodes)
end

function parse_title_from_html(filepath::String)
  html_content = read(filepath, String)
  html = parsehtml(html_content)
  # Select the title element using Cascadia
  title_selector = Selector("head title")
  title_element = eachmatch(title_selector, html.root)

  # Extract the title text
  if !isempty(title_element)
    title = nodeText(title_element[1])
  else
    title = ""
  end
  return title
end



function hfun_taglist_pluto()::String
  tag = locvar(:fd_tag)::String

  c = IOBuffer()
  write(c, "<ul>")

  rpaths = globvar("fd_tag_pages")[tag]
  sorter(p) = begin
    pvd = pagevar(p, "date")
    if isnothing(pvd)
      return Date(Dates.unix2datetime(stat(p).ctime))
    end
    return pvd
  end
  sort!(rpaths, by=sorter, rev=true)



  for rpath in rpaths
    title = parse_title_from_html(replace(rpath, "assets" => "_assets"))

    if isempty(title)
      title = splitext(splitpath(rpath)[end])[1]
      # title = join(nb_name(rpath),".jl")
    end

    url = get_url(rpath)

    write(c, "<li><a href=\"/$rpath\">$title</a></li>")
  end
  write(c, "</ul>")
  return String(take!(c))
end




"""
Working!
Add tags to Franklin variable.
"""
function hfun_get_pluto_tags()
  html_files = get_html_files()

  # Grab all tag names first
  tags = []
  rpaths = []
  for html_file in html_files
    push!(rpaths,replace(html_file, "_assets" => "assets"))
    itag = Franklin.refstring.(parse_tags_from_html(html_file))
    push!(tags, itag)
  end
  frmt_tags = Franklin.DTAG(zip(rpaths,map(Set,tags)))
  Franklin.set_var!(Franklin.GLOBAL_VARS, "fd_page_tags", frmt_tags; check=false)

  return ""
end


"""
Use to add tags on side of page.
"""
function hfun_get_tags()

  tags = Set{String}()

  html_files = get_html_files()

  for html_file in html_files
    file_tags = parse_tags_from_html(html_file)
    union!(tags, file_tags)
  end

  # Sort and collect the tags
  sorted_tags = sort(collect(tags))


  # Generate the HTML for the tag list
  html_buffer = IOBuffer()
  for tag in sorted_tags
    frmt_tag = join([titlecase(word) for word in split(tag)], " ")
    write(
      html_buffer,
      """
<li class="tag-item">
  <a class="tag-link" href="/tag/$(Franklin.refstring(tag))/">$(frmt_tag)</a>
</li>
"""
    )
  end

  return String(take!(html_buffer))
end

2 replies

stefanbringuier May 9, 2023
Author

Works! Thanks a bunch.

I'll use this branch going forward, but would there be any future plans to merge the tags-i1010 into main?

tlienart May 9, 2023
Collaborator

Yes I was waiting for you to confirm that things work fine and ill merge the changes in main now

Using tags from Pluto notebook metadata #1010

Uh oh!

stefanbringuier Apr 20, 2023

Replies: 4 comments · 8 replies

Uh oh!

Uh oh!

tlienart Apr 21, 2023 Collaborator

Uh oh!

stefanbringuier Apr 21, 2023 Author

Uh oh!

Uh oh!

tlienart Apr 22, 2023 Collaborator

Uh oh!

stefanbringuier Apr 24, 2023 Author

Uh oh!

tlienart Apr 24, 2023 Collaborator

Uh oh!

stefanbringuier Apr 24, 2023 Author

Uh oh!

stefanbringuier Apr 28, 2023 Author

Uh oh!

tlienart May 5, 2023 Collaborator

Uh oh!

stefanbringuier May 8, 2023 Author

Uh oh!

Uh oh!

tlienart May 9, 2023 Collaborator

Uh oh!

stefanbringuier May 9, 2023 Author

Uh oh!

tlienart May 9, 2023 Collaborator

stefanbringuier
Apr 20, 2023

Replies: 4 comments 8 replies

tlienart
Apr 21, 2023
Collaborator

stefanbringuier Apr 21, 2023
Author

tlienart
Apr 22, 2023
Collaborator

stefanbringuier Apr 24, 2023
Author

tlienart Apr 24, 2023
Collaborator

stefanbringuier Apr 24, 2023
Author

stefanbringuier Apr 28, 2023
Author

tlienart
May 5, 2023
Collaborator

stefanbringuier May 8, 2023
Author

tlienart
May 9, 2023
Collaborator

stefanbringuier May 9, 2023
Author

tlienart May 9, 2023
Collaborator