Help wanted - YOURLS import error 26

[franco-utalk]

Shlink version

Server: v4.4.6 - Client: v4.4.0

PHP version

8.4.5

How do you serve Shlink

Docker image

Database engine

MySQL

Database version

8.0

Summary

I get the following error when I'm trying to import from YOURL'S

`Exception trace:
  at /etc/shlink/vendor/shlinkio/shlink-importer/src/Http/InvalidRequestException.php:26
 Shlinkio\Shlink\Importer\Http\InvalidRequestException::fromResponseData() at /etc/shlink/vendor/shlinkio/shlink-importer/src/Http/RestApiConsumer.php:37
 Shlinkio\Shlink\Importer\Http\RestApiConsumer->callApi() at /etc/shlink/vendor/shlinkio/shlink-importer/src/Sources/Yourls/YourlsImporter.php:109
 Shlinkio\Shlink\Importer\Sources\Yourls\YourlsImporter->callYourlsApi() at /etc/shlink/vendor/shlinkio/shlink-importer/src/Sources/Yourls/YourlsImporter.php:86
 Shlinkio\Shlink\Importer\Sources\Yourls\YourlsImporter->loadVisits() at /etc/shlink/module/Core/src/Util/DoctrineBatchHelper.php:31
 Shlinkio\Shlink\Core\Util\DoctrineBatchHelper->wrapIterable() at /etc/shlink/module/Core/src/Importer/ShortUrlImporting.php:46
 Shlinkio\Shlink\Core\Importer\ShortUrlImporting->importVisits() at /etc/shlink/module/Core/src/Importer/ImportedLinksProcessor.php:86
 Shlinkio\Shlink\Core\Importer\ImportedLinksProcessor->importShortUrls() at /etc/shlink/module/Core/src/Importer/ImportedLinksProcessor.php:43
 Shlinkio\Shlink\Core\Importer\ImportedLinksProcessor->process() at /etc/shlink/vendor/shlinkio/shlink-importer/src/Command/ImportCommand.php:84
 Shlinkio\Shlink\Importer\Command\ImportCommand->execute() at /etc/shlink/vendor/symfony/console/Command/Command.php:279
 Symfony\Component\Console\Command\Command->run() at /etc/shlink/vendor/symfony/console/Application.php:1076
 Symfony\Component\Console\Application->doRunCommand() at /etc/shlink/vendor/symfony/console/Application.php:342
 Symfony\Component\Console\Application->doRun() at /etc/shlink/vendor/symfony/console/Application.php:193
 Symfony\Component\Console\Application->run() at /etc/shlink/bin/cli:10`

An ideas, I have increased memory for PHP to 512 but always stops at the same spot.

Could it be due to the number of visits for sweatcoin url link = 518646. It does state on the plugin page the following

**If you have a lot of links, or some link with a lot of visits, that could result in an error either on YOURLS or Shlink.

The behaviour will depend on how much memory your instances are allowed to use.**

My current list is around 640 links, is there a way that I can import the CSV so it does the visits etc like it does with the plugin. The documentation for CSV import does not mention that.

[acelaya]

This error basically means your YOURLS instance is erroring out, so you should see their logs to get more details.

In any case, the problem is very likely the 518646 short URLs. YOURLs does not expose an API, so I created a plugin to allow importing URLs from it https://github.com/acelaya/yourls-shlink-import-plugin

Sadly, this plugin has a limitation, which is that it tries to load all links at once, with no pagination, so that is very likely a problem with so many links.

If you want to give it a go at supporting pagination, it would require changing this code https://github.com/acelaya/yourls-shlink-import-plugin/blob/420fdd9/plugin.php#L14-L21, to read pagination query params. Then the response would also need to include the amount of pages.

Then the logic to make requests to YOURLS is here https://github.com/shlinkio/shlink-importer/blob/872ba26/src/Sources/Yourls/YourlsImporter.php#L65. That would need to be expanded to include the pagination query params, and recursively make more requests until all pages have been processed. The ShlinkImporter works like this if you want to check.

[acelaya]

I just realized this has been in my TODO list since 2021 😅 acelaya/yourls-shlink-import-plugin#1