Hi Saul-Mirone, I used @hocuspocus/server as milkdown collab-server to save doc to database

[yswang0927]

Hi @Saul-Mirone ,

I used @hocuspocus/server as milkdown collab-server to save doc to database,
and I also want to get the plain markdown content from yjs-Doc in the server, how to do it?

Thank you!

The client code:

const wsUrl = 'ws://127.0.0.1:2345';
const wsRoom = 'milkdown';

this.doc = new Doc();

this.wsProvider = new HocuspocusProvider({
    url: wsUrl,
    name: wsRoom,
    document: this.doc,
    connect: true,
    onDisconnect(data) {
        console.log('>>> onDisconnect: ', data);
    },
});

this.wsProvider.setAwarenessField("user", {"name":"Emma", "color":"#ff0000"});

this.collabService.bindDoc(this.doc).setAwareness(this.wsProvider.awareness!);

this.wsProvider.on('synced', (data: {state: boolean}) => {
    if (data && data.state) {
        //this.collabService.applyTemplate(template).connect();
        this.collabService.connect();
    }
});

The server code:

import { Server } from "@hocuspocus/server";
import { Database } from "@hocuspocus/extension-database";
import { Doc } from "yjs";
import pg from 'pg';

const dbPool = new pg.Pool({
  host: '127.0.0.1',
  port: 5432,
  user: 'postgres',
  password: 'postgres@123',
  database: 'postgres',
  ssl: false,
  max: 20,
  min: 2,
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 5000,
  maxLifetimeSeconds: 3600,
});

// store and read milkdown yjs data to postgresql db
const pgExtension = new Database({
  fetch: async ({documentName}) => {
    const sql = "SELECT data FROM yjs_data WHERE doc_id = $1";
    const sqlParams = [documentName];

    const result = await dbPool.query(sql, sqlParams);
    if (result.rows.length > 0) {
      return result.rows[0].data;
    }
    
    return null;
  },

  /**
   * @param document  yjs-Doc
   * @param state yjs-Doc state
   */
  store: async ({documentName, document, state}) => {
    const sql = "INSERT INTO yjs_data(doc_id, data) VALUES($1, $2) ON CONFLICT(doc_id) DO UPDATE SET data = EXCLUDED.data";
    const sqlParams = [documentName, state];

    try {
      const result = await dbPool.query(sql, sqlParams);
      console.log('>>> save-to-db = ', result.rowCount);
    } catch (e) {
      console.error('>>> failed-save-to-db ', e);
      throw e;
    }
  }
});

pgExtension.onConfigure = async (data) => {
  const table_ddl = `CREATE TABLE IF NOT EXISTS public.yjs_data (
    "doc_id" varchar(255) NOT NULL,
    "data" bytea NULL,
    CONSTRAINT yjs_data_pk PRIMARY KEY (doc_id)
  )`;
  const result = await dbPool.query(table_ddl);
  console.log('>>> create-table-if-notexists: ', result);
};


const server = Server.configure({
  name: 'milkdown-collab-server',
  address: '0.0.0.0',
  port: 2345,
  timeout: 30000,
  debounce: 5000,
  maxDebounce: 10000,
  quiet: false,

  async onDestroy(data) {
    dbPool.end(() => {
      console.log('>>> Node-PG pool shutdown!');
    });
    console.log(`>>> Server was shutdown!`);
  },

  extensions: [pgExtension]

});

server.listen();

The server code store: async ({documentName, document, state}) => {} invoked to save milkdown yjs data to db,

• the argument document is yjs-Doc object,
• the argument state is yjs-Doc state binary data,

how to use milkdown-parser or yjs-parser convert the document or state data to plain markdown content ?

I only can do this:

import { yXmlFragmentToProseMirrorRootNode } from 'y-prosemirror';

store: async ({documentName, document, state}) => {
    const xmlFragment = document.getXmlFragment('prosemirror');
    const yDocNode = yXmlFragmentToProseMirrorRootNode(xmlFragment, schema);

    // ... How to parser `yDocNode` to markdown content?

}

Thank you!

[yswang0927]

@Saul-Mirone , I found your code use milkdown in nodejs: https://stackblitz.com/edit/node-ltifxb?file=index.js

and I update the code, it can parse to markdown-content, but it not supports task [ x ] , image, Latex, table, ... , even I add commonmark-schema and gfm-schema.

import { yXmlFragmentToProseMirrorRootNode } from 'y-prosemirror';

import {
  init,
  InitReady,
  initTimerCtx,
  remarkCtx,
  schema,
  schemaCtx,
  SchemaReady,
  serializerCtx
} from '@milkdown/core';
import { Clock, Container, Ctx } from '@milkdown/ctx';
import { schema as commonSchema } from '@milkdown/preset-commonmark';
import { schema as gfmSchema } from '@milkdown/preset-gfm';
import { ParserState, SerializerState } from '@milkdown/transformer';

import { TimerType } from './fake-timer.js';

store: async ({ document }) => {
    const clock = new Clock();
    const container = new Container();
    const ctx = new Ctx(container, clock);
    const runInit = init({})(ctx);
    const runSchema = schema(ctx);

    const hackTimer = (name, target) => {
      const fake = new TimerType(name);
      fake.id = target.id;
      const result = fake.create(clock.store);
      clock.store.set(target.id, result);
      return result;
    };
    
    hackTimer('InitReady', InitReady);
    hackTimer('SchemaReady', SchemaReady);
    
    ctx.set(initTimerCtx, []);

    // I add commonmark-schema and gfm-schema
    const allSchemas = [commonSchema, gfmSchema].flat();

    await Promise.all([
      runInit(),
      runSchema(),
      ...allSchemas.map((node) => node(ctx)()),
    ]);
  
    const schema2 = ctx.get(schemaCtx);
    const remark = ctx.get(remarkCtx);
    const serializer = SerializerState.create(schema2, remark);

    const xmlFragment = document.getXmlFragment('prosemirror');
    const pmNode = yXmlFragmentToProseMirrorRootNode(xmlFragment, schema2);
    let markdownContent = serializer(pmNode);
    console.log('>> markdownContent: ', markdownContent);

  }