Get all HubSpot properties

[erummel]

Source name

hubspot

Describe the data you'd like to see

dlt's HubSpot source connector limits the number of properties it can ingest: a comma-separated, whitespace-free list of desired property names may not exceed 2,000 characters in length. Meanwhile, I confirmed that Airbyte ingests all properties (except 5 or so deprecated ones).

Would it be feasible for dlt to get all HubSpot properties, standard and custom, by default?

---

To clarify some of my responses to the dropdowns:

• I'm considering using dlt, but this lack of functionality, in part, prevents this.
• I can't currently contribute this extension. That may change.

Are you a dlt user?

I'm considering using dlt, but this lack of functionality, in part, prevents this.

Are you ready to contribute this extension?

No.

dlt destination

Azure Blob Storage (as Delta Lake tables)

Additional information

No response

[cmtk]

I was able to get around this by increasing the MAX_PROPS_LENGTH = 2000 value in hubspot/settings.py to a much higher value (12000 in my case).

Not sure why it is defaulted relatively low, I couldn't find anything in the Hubspot docs discussing a limit. Although it is possible certain endpoints have lower limits. By raising this, you may eventually hit the Hubspot server's HTTP GET length limit, but one person on the Hubspot forums reported that may be as high as 16300 characters, total. Not an official source, but all I found so far.

About the limit, the GET limit is actually defined between client and server, the 2048 limit is a defaut for some browsers, but in this case I was able to get about 16300 characters without an error, so that isn't the limit.

https://community.hubspot.com/t5/APIs-Integrations/Maximum-length-of-GET-request-for-the-API/m-p/300370/highlight/true#M28238

Raising MAX_PROPS_LENGTH worked for me with an 11940 symbol long prop list.

Now the fact that I have that many in the first place is a separate problem to solve outside of dlt. It's aggravated by the fact that Hubspot allows and auto-creates up to 100-character-long property keys based on the title.

This is likely still a valid feature request for chunking the property list when over a certain length. It looks like this may already be partially handled in some other places:

verified-sources/sources/hubspot/__init__.py

Lines 233 to 241 in f3a8603

	# Since the length of request should be less than MAX_PROPS_LENGTH, we cannot request
	# data for the whole properties list. Therefore, in the following lines we request
	# data iteratively for chunks of the properties list.
	for chunk in chunk_properties(date_entered_properties, MAX_PROPS_LENGTH):
	props_part = ",".join(chunk)
	for data in fetch_data_for_properties(
	props_part, api_key, object_type, soft_delete
	):
	yield pivot_stages_properties(data)

[erummel]

When I ask HubSpot for all, e.g., company properties, it returns 234 of them:

import os
from hubspot import HubSpot

def get_properties(obj_name: str) -> list[str]:
    client = HubSpot(access_token=os.environ["HUBSPOT_ACCESS_TOKEN"])
    res = client.crm.properties.core_api.get_all(object_type=obj_name)
    return [obj["name"] for obj in res.to_dict()["results"]]

That's 234 properties I want to ingest. Even if I set dlt's HubSpot source's MAX_PROPS_LENGTH to 12000, 16300, or something ridiculous like 500000, dlt only ingests 72 (ignoring dlt-generated metadata).

Of those, 27 aren't ingested because they didn't receive any data from our little sandbox HubSpot env, and I see that #643 covers this. So that's fine.

That leaves 135 properties that dlt misses for no given reason. All of them begin with hs_*.

Update 2025-09-08: Today I could get 107 properties using the dlt init-provided load_crm_data() function, instead of load_crm_data_with_history(), which is what I used before.