Meeting migration question

[kevindherman]

If a current ProudCity customer is setting up a new instance and wants to bring over two years of meetings material is the below migration possible:

1. Export meetings content type current site
2. Import two years into the new instance
3. Move uploaded PDFs/files from current site into media library of new site to complete the migration

Would step three need to do some python google-cli fu to pattern match the filenames and upload a new copy if they match as mentioned in this issue: https://github.com/proudcity/pc-dev-issues/issues/142

Here is an example of a type of meeting filled with uploaded and hyperlinked (to media files): https://www.cityofsanrafael.org/meetings/city-council-august-18-2025/

More examples here: https://www.cityofsanrafael.org/city-council-meetings/

Tasks

• can migrate DB pro pull a CPT and then integrate it with the existing content
• can WP All Import pull a CPT with media and integrate it into the site

[curtismchale]

@kevindherman yes this is possible. It may cost us a bit for a plugin ($299 for this option) but we may also be able to do it with free tools. I'd have to run a test or two to confirm which options will work.

There is no pattern match stuff needed as we're not replacing the media files in place, but creating a new set of folders for them.

[curtismchale]

@kevindherman I've taken a bigger look into this and we do use the attachment_id in our templates to see if a file exists. We then use it to build some data, but I don't see a reason we do it that way. The data url is stored in the database so we can check if that URL exists and if it does we output our download buttons and tabs.

I have to think about it some more so this should be a discussion for tomorrow.

I currently see 2 options.

1. We manually move years worth of meetings as previously discussed. I can get enough of the data in that it's just going to be uploading the files in question for Agenda's
2. I modify the templates for everyone so that it's not looking for the attachment_id and then we don't have to move the files.

[kevindherman]

@curtismchale sounds good. Lets discuss tomorrow and hopefully that's enough for me to get back to the customer on scoping.

[curtismchale]

Looking further we use the attachment_id to get various attachment metadata and then display information on filesize, filetype and other metadata. But I could also get a script together that goes back through the imported media for meetings and adds it to the media gallery in the new location which would mean we don't modify your current templates and we still get the files migrated to the new media gallery.

There are still unknowns though. It's possible the plugin takes longer than anticipated depending on how our servers block automated downloads. I don't think this will be an issue due to previous work downloading PDF's for other projects via terminal, but I'm not 100% sure.

[curtismchale]

@kevindherman So yes we can move meetings by year. The extra cost per year is minimal now that we've done the upfront work to figure it out.

First we use WP All Import/Export to grab the meetings we want. Here is where we limit this by year to whatever timeframe we want. There is a bit of extra time if we include more years in that the export/import takes longer because there are more entries.

Once the meetings are up on the new location we can use the plugin I wrote to migrate the uploaded files to the new location. This does not move files that are simply linked in content. It does move files added via the upload buttons for meeting minutes, agenda, agenda packets. It changes their URL and puts them in the media gallery. This takes time as it depends on the memory of the server. We can run around 30 meetings at a time and there is a 30 - 60 second wait between each run. I can increase the memory for a limited time to allow bigger runs but this is the most time intensive part of the process.

To "move" the files that are linked in content I'd need to do a search/replace on part of the URL string. I haven't tried this yet because it would also need the next step. I'd search for the old location across the site and then move it to the new location.

Files that have been simply linked to in the content of a meeting will need to be moved manually as previously discussed. I can do some Google CLI fu to download the existing media from the site. Then the best step is to download the new media folders from the beta site and integrate them locally. Once integrated I can upload the new combined media folder to the server and swap it with the old one.

Time spent

• 5 - 8ish hours so far to figure this out
• call it 2 hours extra per year of meetings due to the button clicking to migrate files
• 2ish hours to do the media integration. This doesn't really change if there are more years of meetings
• 1 - 2 hours for testing of the search/replace just in case there are issues as there often are with edge cases

[kevindherman]

@curtismchale to make sure I understand the 2-4ish hours for media integration and search/replace is what you anticipate for dealing with the media that is linked in content, correct?

I know you mentioned the amount of years doesn't really change anything but this format is really just used for council meetings and there are about 100 going back to Jan 2023: https://www.cityofsanrafael.org/wp-admin/edit.php?meeting-taxonomy=city-council-successor-agency&post_type=meeting&mode=list

One more question: does the migration bring over the meeting categories as well? I'm assuming yes, but want to confirm.

This is super helpful. Once the above confirmed, I'll have what I need.

[curtismchale]

@kevindherman

Yes it brings the taxonomies over and adds the meetings to them. You can take a look through Beta to see this. I only clicked the button to migrate around the first 40 meetings so you won't see linked PDF downloads for some of the meetings I imported.

[kevindherman]

@curtismchale and is my assumption correct on the time part (the first question above)?

[kevindherman]

Migration looks great from the file perspective which is definitely the hardest part. Using these as a reference here is some feedback for when this moves forward:

1. BETA version
2. Live version

Feedback:

• YouTube video field is missing from migration
• Contact is missing from migration (this might not work since it relies on selecting a department). This would be nice since the departments will be setup on the new site to match to but it's probably the least important out of the above.

[curtismchale]

We don't have a YouTube video because I pulled a local copy of the site and was doing the export there. Due to our pods this is the way for the export to work. The YouTube video wasn't added to the meeting when I pulled a copy of it. See this October meeting to see the YouTube video intact. If the data is in the meeting already then it will show up.

The agency data is there but it's stored as a post_id which means it will change on the new site. If we want that data we can either port it manually to the new site or I'd need to build a plugin to loop through all the stored agency_id fields and get the Agency Slug then save that alongside the meeting. As long as the agency had the exact same title/slug on the new site we could then reverse that process to map them to the new agency_id.

It's really a matter of how many meetings we'd be doing to know if it's worthwhile. If it's 100 meetings, it's not worth writing the plugin. If it's 500 meetings (which is what Jan 1 2023 and newer gets us) then it's likely worth it. It would also give us the ability to do this in the future for a customer.