Embed plugin not working on production

[driesvints]

I have the weirdest issue where the Embed plugin is not transforming a YouTube link into an embed on production. Works perfectly locally with the same content.

The url is https://youtube.com/watch?v=WZS12h2m0uI
Config (config('app.host') returns the domain of the site)

use League\CommonMark\Environment\Environment;
use League\CommonMark\Extension\CommonMark\CommonMarkCoreExtension;
use League\CommonMark\Extension\Embed\Bridge\OscaroteroEmbedAdapter;
use League\CommonMark\Extension\Embed\EmbedExtension;
use League\CommonMark\Extension\ExternalLink\ExternalLinkExtension;
use League\CommonMark\Extension\GithubFlavoredMarkdownExtension;
use League\CommonMark\Extension\Mention\MentionExtension;
use League\CommonMark\MarkdownConverter;

$environment = new Environment([
    'html_input' => 'escape',
    'max_nesting_level' => 10,
    'allow_unsafe_links' => false,
    'mentions' => [
        'username' => [
            'prefix' => '@',
            'pattern' => '[a-z\d](?:[a-z\d]|-(?=[a-z\d])){0,38}(?!\w)',
            'generator' => config('app.url').'/user/%s',
        ],
    ],
    'external_link' => [
        'internal_hosts' => config('app.host'),
        'open_in_new_window' => true,
        'nofollow' => '',
    ],
    'embed' => [
        'adapter' => new OscaroteroEmbedAdapter,
        'allowed_domains' => ['youtube.com', 'twitter.com'],
        'fallback' => 'link',
    ],
]);

$environment->addExtension(new CommonMarkCoreExtension);
$environment->addExtension(new GithubFlavoredMarkdownExtension);
$environment->addExtension(new MentionExtension);
$environment->addExtension(new ExternalLinkExtension);
$environment->addExtension(new EmbedExtension);

new MarkdownConverter($environment);

I can't for the life of me fathom why this is happening. It's not region bound because I checked from some VPS sources. The iframe simply isn't rendered. It's also (afaik) not caching. Any idea why the Embed plugin might not render it?

[colinodell]

My best guess is that the underlying Embed library is probably not getting the expected response back for that URL for some reason. If you're able to set a breakpoint or add some logging here to inspect the value of $extractor you might get some answers.

On a related note, we could probably do a better job handling errors in our adapter. But it's not clear to me whether/how the Embed library actually exposes those 🤔

[driesvints]

Thanks a bunch for helping out @colinodell. I tried logging those values but gotten no results back so thinking the root cause is somewhere deeper. I'll try overriding some more classes and adding some more logging later on.

[driesvints]

I managed to get to the bottom of this, thanks for the hint of $extractor @colinodell. It seems it's a "Too many requests issue". Here's the full logs:

Array ( [0] => Embed\Extractor Object ( [request:Embed\Extractor:private] => GuzzleHttp\Psr7\Request Object ( [method:GuzzleHttp\Psr7\Request:private] => GET [requestTarget:GuzzleHttp\Psr7\Request:private] => [uri:GuzzleHttp\Psr7\Request:private] => GuzzleHttp\Psr7\Uri Object ( [scheme:GuzzleHttp\Psr7\Uri:private] => https [userInfo:GuzzleHttp\Psr7\Uri:private] => [host:GuzzleHttp\Psr7\Uri:private] => youtube.com [port:GuzzleHttp\Psr7\Uri:private] => [path:GuzzleHttp\Psr7\Uri:private] => /watch [query:GuzzleHttp\Psr7\Uri:private] => v=WZS12h2m0uI [fragment:GuzzleHttp\Psr7\Uri:private] => [composedComponents:GuzzleHttp\Psr7\Uri:private] => https://youtube.com/watch?v=WZS12h2m0uI ) [headers:GuzzleHttp\Psr7\Request:private] => Array ( [Host] => Array ( [0] => youtube.com ) [User-Agent] => Array ( [0] => Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:73.0) Gecko/20100101 Firefox/73.0 ) [Cache-Control] => Array ( [0] => max-age=0 ) ) [headerNames:GuzzleHttp\Psr7\Request:private] => Array ( [host] => Host [user-agent] => User-Agent [cache-control] => Cache-Control ) [protocol:GuzzleHttp\Psr7\Request:private] => 1.1 [stream:GuzzleHttp\Psr7\Request:private] => ) [response:Embed\Extractor:private] => GuzzleHttp\Psr7\Response Object ( [reasonPhrase:GuzzleHttp\Psr7\Response:private] => Too Many Requests [statusCode:GuzzleHttp\Psr7\Response:private] => 429 [headers:GuzzleHttp\Psr7\Response:private] => Array ( [location] => Array ( [0] => https://www.google.com/sorry/index?continue=https://youtube.com/watch%3Fv%3DWZS12h2m0uI&q=EgSkXOEgGJbDwcEGIjC311hZXbJr5FSI1XiT1XPfyc0fHje7uaSTXLmTp7V_BwVClM-F5Lwuw-ASG6ZU1lIyAnJSWgFD ) [date] => Array ( [0] => Fri, 23 May 2025 11:52:54 GMT [1] => Fri, 23 May 2025 11:52:54 GMT ) [pragma] => Array ( [0] => no-cache [1] => no-cache ) [expires] => Array ( [0] => Fri, 01 Jan 1990 00:00:00 GMT [1] => Fri, 01 Jan 1990 00:00:00 GMT ) [cache-control] => Array ( [0] => no-store, no-cache, must-revalidate [1] => no-store, no-cache, must-revalidate ) [content-type] => Array ( [0] => text/html; charset=UTF-8 [1] => text/html ) [server] => Array ( [0] => HTTP server (unknown) [1] => HTTP server (unknown) ) [content-length] => Array ( [0] => 383 [1] => 3160 ) [x-xss-protection] => Array ( [0] => 0 [1] => 0 ) [alt-svc] => Array ( [0] => h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 HTTP/2 429 [1] => h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 ) [Content-Location] => Array ( [0] => https://www.google.com/sorry/index?continue=https://youtube.com/watch%3Fv%3DWZS12h2m0uI&q=EgSkXOEgGJbDwcEGIjC311hZXbJr5FSI1XiT1XPfyc0fHje7uaSTXLmTp7V_BwVClM-F5Lwuw-ASG6ZU1lIyAnJSWgFD ) [X-Request-Time] => Array ( [0] => 0.502 ms ) ) [headerNames:GuzzleHttp\Psr7\Response:private] => Array ( [location] => location [date] => date [pragma] => pragma [expires] => expires [cache-control] => cache-control [content-type] => content-type [server] => server [content-length] => content-length [x-xss-protection] => x-xss-protection [alt-svc] => alt-svc [content-location] => Content-Location [x-request-time] => X-Request-Time ) [protocol:GuzzleHttp\Psr7\Response:private] => 1.1 [stream:GuzzleHttp\Psr7\Response:private] => GuzzleHttp\Psr7\Stream Object ( [stream:GuzzleHttp\Psr7\Stream:private] => Resource id #21 [size:GuzzleHttp\Psr7\Stream:private] => [seekable:GuzzleHttp\Psr7\Stream:private] => 1 [readable:GuzzleHttp\Psr7\Stream:private] => 1 [writable:GuzzleHttp\Psr7\Stream:private] => 1 [uri:GuzzleHttp\Psr7\Stream:private] => php://temp [customMetadata:GuzzleHttp\Psr7\Stream:private] => Array ( ) ) ) [uri:Embed\Extractor:private] => GuzzleHttp\Psr7\Uri Object ( [scheme:GuzzleHttp\Psr7\Uri:private] => https [userInfo:GuzzleHttp\Psr7\Uri:private] => [host:GuzzleHttp\Psr7\Uri:private] => www.google.com [port:GuzzleHttp\Psr7\Uri:private] => [path:GuzzleHttp\Psr7\Uri:private] => /sorry/index [query:GuzzleHttp\Psr7\Uri:private] => continue=https://youtube.com/watch%3Fv%3DWZS12h2m0uI&q=EgSkXOEgGJbDwcEGIjC311hZXbJr5FSI1XiT1XPfyc0fHje7uaSTXLmTp7V_BwVClM-F5Lwuw-ASG6ZU1lIyAnJSWgFD [fragment:GuzzleHttp\Psr7\Uri:private] => [composedComponents:GuzzleHttp\Psr7\Uri:private] => https://www.google.com/sorry/index?continue=https://youtube.com/watch%3Fv%3DWZS12h2m0uI&q=EgSkXOEgGJbDwcEGIjC311hZXbJr5FSI1XiT1XPfyc0fHje7uaSTXLmTp7V_BwVClM-F5Lwuw-ASG6ZU1lIyAnJSWgFD ) [crawler:Embed\Extractor:private] => App\Crawler Object ( [defaultHeaders:Embed\Http\Crawler:private] => Array ( [User-Agent] => Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:73.0) Gecko/20100101 Firefox/73.0 [Cache-Control] => max-age=0 ) [requestFactory:protected] => GuzzleHttp\Psr7\HttpFactory Object ( ) [uriFactory:protected] => GuzzleHttp\Psr7\HttpFactory Object ( ) [client:protected] => Embed\Http\CurlClient Object ( [responseFactory:Embed\Http\CurlClient:private] => GuzzleHttp\Psr7\HttpFactory Object ( ) [settings:Embed\Http\CurlClient:private] => Array ( ) ) [defaultHeaders:protected] => Array ( [User-Agent] => Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:73.0) Gecko/20100101 Firefox/73.0 [Cache-Control] => max-age=0 ) ) [document:protected] => Embed\Document Object ( [extractor:Embed\Document:private] => Embed\Extractor Object *RECURSION* [document:Embed\Document:private] => DOMDocument Object ( [doctype] => (object value omitted) [implementation] => (object value omitted) [documentElement] => (object value omitted) [actualEncoding] => utf-8 [encoding] => utf-8 [xmlEncoding] => utf-8 [standalone] => 1 [xmlStandalone] => 1 [version] => [xmlVersion] => [strictErrorChecking] => 1 [documentURI] => [config] => [formatOutput] => [validateOnParse] => [resolveExternals] => [preserveWhiteSpace] => 1 [recover] => [substituteEntities] => [firstElementChild] => (object value omitted) [lastElementChild] => (object value omitted) [childElementCount] => 1 [nodeName] => #document [nodeValue] => [nodeType] => 13 [parentNode] => [parentElement] => [childNodes] => (object value omitted) [firstChild] => (object value omitted) [lastChild] => (object value omitted) [previousSibling] => [nextSibling] => [attributes] => [isConnected] => 1 [ownerDocument] => [namespaceURI] => [prefix] => [localName] => [baseURI] => [textContent] => https://youtube.com/watch?v=WZS12h2m0uI In order to continue, please enable javascript on your web browser. var submitCallback = function(response) {document.getElementById('captcha-form').submit();}; About this page Our systems have detected unusual traffic from your computer network. This page checks to see if it's really you sending the requests, and not a robot. Why did this happen? This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the Terms of Service. The block will expire shortly after those requests stop. In the meantime, solving the above CAPTCHA will let you continue to use our services. This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests. If you share your network connection, ask your administrator for help — a different computer using the same IP address may be responsible. Learn more Sometimes you may be asked to solve the CAPTCHA if you are using advanced terms that robots are known to use, or sending requests very quickly. IP address: 164.92.225.32 Time: 2025-05-23T11:52:54Z URL: https://youtube.com/watch?v=WZS12h2m0uI ) [xpath:Embed\Document:private] => DOMXPath Object ( [document] => (object value omitted) [registerNodeNamespaces] => 1 ) ) [oembed:protected] => Embed\OEmbed Object ( [defaults:Embed\OEmbed:private] => Array ( ) [endpoint:Embed\OEmbed:private] => [extractor:protected] => Embed\Extractor Object *RECURSION* [data:Embed\OEmbed:private] => Array ( ) ) [linkedData:protected] => Embed\LinkedData Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [metas:protected] => Embed\Metas Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [settings:Embed\Extractor:private] => Array ( ) [customDetectors:Embed\Extractor:private] => Array ( ) [authorName:protected] => Embed\Detectors\AuthorName Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [authorUrl:protected] => Embed\Detectors\AuthorUrl Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [cms:protected] => Embed\Detectors\Cms Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [code:protected] => Embed\Detectors\Code Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [description:protected] => Embed\Detectors\Description Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [favicon:protected] => Embed\Detectors\Favicon Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [feeds:protected] => Embed\Detectors\Feeds Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [icon:protected] => Embed\Detectors\Icon Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [image:protected] => Embed\Detectors\Image Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [keywords:protected] => Embed\Detectors\Keywords Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [language:protected] => Embed\Detectors\Language Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [languages:protected] => Embed\Detectors\Languages Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [license:protected] => Embed\Detectors\License Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [providerName:protected] => Embed\Detectors\ProviderName Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [providerUrl:protected] => Embed\Detectors\ProviderUrl Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [publishedTime:protected] => Embed\Detectors\PublishedTime Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [redirect:protected] => Embed\Detectors\Redirect Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* [cache:Embed\Detectors\Detector:private] => Array ( [cached] => 1 [value] => ) ) [title:protected] => Embed\Detectors\Title Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) [url:protected] => Embed\Detectors\Url Object ( [extractor:protected] => Embed\Extractor Object *RECURSION* ) ) )

[driesvints]

I do wonder why I'm not getting it locally and only in production 🤔