Issues trying to use EmbedV4 from Cohere hosted on Azure AI Foundry

[baptchv]

I have deployed Cohere EmbedV4 on Azure but have a hard time making it work. I am using python 3.11, and azure.ai.inference package.
Here is my code sample :

client = EmbeddingsClient(
    endpoint="https://<myProject>.services.ai.azure.com/models",
    credential=AzureKeyCredential("myKey")
) 

response = client.embed(
    input=["first phrase","second phrase","third phrase"],
    model = "embed-v-4-0-2",
)

I am getting this error :

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Which doesn't tell much. Looking deeper and trying to debug the lib it may look like an api_version issue ? But i'm really not sure and would really appreciate an expert eye on this matter.

Note that :

• Cohere3.5 isn't working.
• But other embeddings like ada-002 works fine.
• I've tried to redeploy multiple times.
• I've waited 72h+ to be sure the model was successfully deployed.

Thanks

[leestott]

Hi, The JSONDecodeError at line 1 column 1 suggests that the response from the API might be empty or improperly formatted. Here are a few things to check:

1. API Version Compatibility – The azure-ai-inference package uses specific API versions. Ensure that your deployment supports the correct version. You might need to explicitly set the api_version parameter when initializing the client.

2. Endpoint & Credentials – Double-check that your endpoint URL is correct and matches the region where your model is deployed. Also, verify that your API key is valid.

3. Model Availability – Since other embeddings like ada-002 work fine, but Cohere EmbedV4 doesn’t, it’s possible that the model deployment is incomplete or misconfigured. You might want to check the Azure AI Foundry dashboard for any deployment errors.

4. Debugging the Response – Try printing the raw response before parsing it:

   response = client.embed(
       input=["first phrase", "second phrase", "third phrase"],
       model="embed-v-4-0-2",
   )
   print(response)

   If the response is empty or contains an error message, that could provide more insight.

5. Azure AI Inference Package Version – Ensure you're using the latest version of azure-ai-inference. You can update it using:

   pip install --upgrade azure-ai-inference
   

Here is a full python example

# Create a function to check the response content for debugging
def debug_response(response_content):
    """Debug the raw API response content."""
    logger.info(f"Response status code: {response_content.status_code}")
    logger.info(f"Response headers: {response_content.headers}")
    
    try:
        content = response_content.text
        logger.info(f"Raw response content: {content[:100]}...")  # Log first 100 chars
        return content
    except Exception as e:
        logger.error(f"Error accessing response content: {str(e)}")
        return None

# Main function to demonstrate embedding with Cohere models
def get_embeddings():
    """
    Generate embeddings using the Cohere EmbedV4 model on Azure AI Platform.
    
    The key issues to check:
    1. Correct API endpoint format (note the /embeddings path for Cohere models)
    2. API version specification (2024-02-01-preview is required for newer models)
    3. Correct model name as deployed in Azure AI
    4. Proper input format and parameters
    """
    
    # Get credentials from environment variables
    endpoint = os.getenv("AZURE_ENDPOINT")
    api_key = os.getenv("AZURE_API")
    model_name = os.getenv("AZURE_EMBED_MODEL", "embed-v-4-0-2")
    
    if not endpoint or not api_key:
        raise ValueError("Missing required environment variables AZURE_ENDPOINT and AZURE_API")
    
    logger.info(f"Using endpoint: {endpoint}")
    logger.info(f"Using model: {model_name}")
    
    # For Cohere models, the endpoint format may need adjustment
    # If the endpoint doesn't already include the correct path
    if not endpoint.endswith('/embeddings'):
        # Check if we need to append /embeddings to the endpoint
        if endpoint.endswith('/'):
            endpoint = f"{endpoint}embeddings"
        else:
            endpoint = f"{endpoint}/embeddings"
        logger.info(f"Adjusted endpoint to: {endpoint}")
    
    # Initialize the embeddings client
    try:
        client = EmbeddingsClient(
            endpoint=endpoint,
            credential=AzureKeyCredential(api_key),
            api_version="2024-02-01-preview"  # Specify API version explicitly
        )
        logger.info("Client initialized successfully.")
    except Exception as e:
        logger.error(f"Error initializing client: {str(e)}")
        raise
    
    # Sample text for embeddings
    texts = [
        "First phrase: Azure AI services provide powerful capabilities.",
        "Second phrase: Embeddings convert text into numerical vectors.",
        "Third phrase: These vectors capture semantic meaning."
    ]
    
    # Optional parameters for Cohere models
    embedding_options = {
        "input_type": "search_document",  # Can be 'search_document', 'search_query', 'classification', 'clustering'
        "truncate": "END"  # How to handle texts exceeding the token limit: 'NONE', 'START', 'END'
    }
    
    try:
        # First attempt - standard way
        logger.info("Attempting to get embeddings...")
        response = client.embed(
            input=texts,
            model=model_name,
            **embedding_options
        )
        
        # Success path
        logger.info("Successfully generated embeddings!")
        logger.info(f"Embedding dimensions: {len(response.embeddings[0])}")
        logger.info(f"Number of embeddings: {len(response.embeddings)}")
        
        # Return first 5 dimensions of first embedding as sample
        sample = response.embeddings[0][:5]
        logger.info(f"Sample embedding (first 5 dimensions): {sample}")
        
        return response.embeddings
    
    except Exception as primary_error:
        logger.error(f"Primary method failed: {str(primary_error)}")
        logger.info("Attempting alternative approaches...")
        
        try:
            # Try a different method if the model name has dots or hyphens
            model_name_alt = model_name.replace('-', '_').replace('.', '_')
            logger.info(f"Trying with alternate model name format: {model_name_alt}")
            
            response = client.embed(
                input=texts,
                model=model_name_alt
            )
            logger.info("Success with alternative model name!")
            return response.embeddings
        
        except Exception as secondary_error:
            logger.error(f"Alternative method also failed: {str(secondary_error)}")
            
            # Final attempt - with minimal parameters
            try:
                logger.info("Final attempt with minimal parameters...")
                response = client.embed(
                    input=texts,
                    model=model_name
                )
                logger.info("Success with minimal parameters!")
                return response.embeddings
            
            except Exception as final_error:
                logger.error(f"All embedding methods failed. Final error: {str(final_error)}")
                
                # Provide guidance on how to fix
                logger.info("\n---- TROUBLESHOOTING GUIDE ----")
                logger.info("1. Check if your model is correctly deployed in Azure AI Foundry")
                logger.info("2. Verify the model name exactly matches the deployment name")
                logger.info("3. Ensure your endpoint format is correct: https://<resource-name>.services.ai.azure.com/embeddings")
                logger.info("4. Check that your API key has proper permissions")
                logger.info("5. Try with api_version='2024-02-01-preview' explicitly")
                logger.info("6. Inspect the Azure AI Foundry deployment logs for any model-specific issues")
                logger.info("7. If possible, try using Azure REST API directly to isolate SDK-specific issues")
                raise RuntimeError("Failed to generate embeddings after multiple attempts") from primary_error

if __name__ == "__main__":
    try:
        embeddings = get_embeddings()
        print(f"Successfully generated {len(embeddings)} embeddings!")
    except Exception as e:
        print(f"Error: {str(e)}")

[baptchv]

Hi Lee, thanks for taking the time.

1. API Version Compatibility – I've tried multiple api_version and it didn't produced better results.

2. Endpoint & Credentials – My model is not deployed in serverless, it is from the model catalog in a services.ai.azure.com endpoint. The region isn't specified but I've copied the exact URI (cf
   
   )

3. Model Availability – No deployment error surfaced.

4. Debugging the Response – Executing client.embed makes the code crash. I can never read the output. The JSON error happens before anything is returned.

5. Azure AI Inference Package Version – Did it + started from a brand new project.

[baptchv]

Running the code you provided returned 404 ressource not found :

Primary method failed: (404) Resource not found
Code: 404
Message: Resource not found
Alternative method also failed: (404) Resource not found
Code: 404
Message: Resource not found
All embedding methods failed. Final error: (404) Resource not found
Code: 404
Message: Resource not found
Error: Failed to generate embeddings after multiple attempts

Even if the deployment appears to be correctly deployed :

[baptchv]

Update:
This feature flag was activated (cf. screenshot), which meant the model was force-deployed on *.services.ai.azure.com. Once disabled, I was able to deploy it correctly.

For older models (like ada-002), it apparently doesn’t matter. For Cohere’s, you should disable it.

I was today years old when I learned this even existed.
I'm leaving the discussion open if this isn't an expected behaviour from Azure. But my problem is solved.
Apologies for taking up your time.

[leestott]

@baptchv

Ok this makes sense, you had this preview feature turned on which switches between models using a single endpoint and authentication, making integration more seamless.

For MaaS (Serverless) ensure this is set to “Off,” which controls whether this deployment feature is enabled.

For reference for others, the preview features are set by selecting the Speaker funnel on https://ai.azure.com

If you encounter model deployment issues or inability to deploy Models as Service (serverless) ensure this feature is turned off.

Reference url is https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/deployments-overview