Winter 2024 Developers Workshop Agenda

[darnjo]

The Winter 2024 RESO Developers Workshop will be held in Seattle, WA, on March 5-6. Please contact dev@reso.org for more information.

Please add suggestions for topics to this thread and/or reference any existing Transport Discussions you would like to see covered at the meeting.

---

Some potential topics...

Definition of Originating and Source System Ids

Each resource in the Data Dictionary usually contains an OriginatingSystemID and SourceSystemID field.

OriginatingSystemName field has the following usage:

OriginatingSystemID is as follows:

These fields are often required in certification to filter data sets on shared systems, so the assumption is that's also the case for data consumers, and perhaps it makes sense to standardize them?

For the OriginatingSystemName and "ID" fields, current usage may be too high to try and change what's currently there.

Other approaches?

RESO maintains a set of "authoritative" organization identifiers using the OUID Resource schema for the orgs it certifies, which are mostly MLSs, technology providers, and pooled platforms.

Perhaps there could be a registry/protocol for looking up or locating certified org data providers? As long as they support RESO Common Format for the OUID (aka UOI Resource), meaning host a JSON file of their orgs matching the DD schema, then data can be sourced from a RESO Web API or or any other https URL that resolves to the canonical format.

However, since the current usage is high, and not standardized (though the fields are intended to be enumerated in the Data Dictionary), it may not make sense to try and change them.

Maybe since RESO had discussed a new Unique Organization Identifier (UOI) in other scenarios, it would make sense to introduce new OriginatingSystemUoi and SourceSystemUoi fields? For the roughly 500 MLSs RESO has in its data set, the information is fairly accurate. Are others willing to publicly host their org info beyond this?

Originating System Names and Ids in Metadata Resources

One thing that doesn't currently exist in the Model, Field, and Lookup Resource proposals is the originating and source system information which is commonly part of the definition of other resources.

However, there is one potential consideration in modeling unique systems in the metadata resources.

In some cases, resources or fields can have different attributes in different systems, such as the ability to search or write data to. In this case, even though the resources and system names might repeat, it may make sense for each item to have duplicate items with their own keys and attributes even though their resource and field names are the same.

Is this also the case for lookup values? Do they have attributes unique to each system beyond being present?

Definition and Enforcement of Related Lookups in the Data Dictionary and Certification

The Fast Track and Certification Analytics subgroups had discussed enforcing related fields / enumeration starting with the most-used first.

So far, this means PropertyType/SubType and StateOrProvince/CountyOrParish fields, going by current usage.

RESO has a past proposal for the RelatedLookup Resource, which allows providers to advertise which fields/enumerations are paired with others. These relationships can either be parent/child siblings.

The Data Dictionary already has definitions for which property types go with which property subtypes, and the fields and enumerations that can be used by both. This is not advertised at the moment in the metadata, but for any data element in the DD, a definition exists.

Potential goals would be to a) formalize these relationships in the RelatedLookup Resource and add certification rules for them, whether providers were using the Model, Field, and Lookup resources or not (~50% of the industry is using the Lookup Resource and 0% for Model and Field resources, so we need a more general option), and b) define ways to advertise this information in the metadata resources.

Rate Limiting

An ongoing topic in Pain Points, which has been confirmed in testing, is that rate limiting considerations have a large impact.

RESO has set two defaults for DD 2.0 testing so far, which can be found here.

• DEFAULT_RATE_LIMITED_WAIT_MINUTES = 60 - The number of numeric minutes to wait after encountering an HTTP 429 error. Default 60m.
• DEFAULT_SECONDS_DELAY_BETWEEN_REQUEST = 1 - The number of numeric seconds to wait between requests in order to avoid rate limiting. Default 1s.

The defaults have been chosen after randomly sampling millions of records from five providers in 400 markets over the course of several months.

The HTTP 429 spec has a retry-after parameter that might be useful here. We discussed this previously in Transport with the Web API Core 2.1.0 spec, but didn't require it. Only the 429 when there are too many requests.

It seems like there should be a way to advertise the quota... in some requests or records per time interval?

It looks like there's an IETF proposal for this but I'm not sure of its current status/usage.

[alifemove]

There was a mention at one point about the way $top works being changed because its used incorrectly? Might be useful to see how its supposed to be done.

[darnjo]

We have coverage for the correct way of doing this (meaning $top with server-driven paging) in the Web API Core 2.1.0 spec and currently allow DD 2.0 certification without Web API Core 2.1.0.

I think the main goal is to get DD 2.0 going now, which is important as it provides a lot of improvements, then providers can start spending time working on addressing this.

[EnFinlay]

Things I'd love to start some conversations around:

1. Replication - not just following nextLinks, but a more compact (avro? ProtBuf? CSV?) way of sharing a very large dataset since JSON is wasteful and making 1000s of requests to page through a whole dataset is error prone. There are many different paths on this topic.
2. Leveraging OData functions for setting sets of fields to return since it's common to run into problems if your $select statement includes more than 2000 characters.
3. This is a little more out there - but in conjunction with the Push Notifications and Subscriptions, can we discuss standards for how that data is consumed / ingested by the entity being pushed to?

[bryanburgers]

And in addition to point 1, is there a way to do an initial fetch of historical data that is better/more efficient for both the producer and the consumer?

Something like a request from the consumer, the producer produces a file that gets placed in s3 or another object store, then communication between the two somehow that the export is done and the file(s) are available at a particular location…

[darnjo]

For replication, I'm guessing that taking on another serialization format when many still struggle with JSON would be challenging. Especially given that Avro and Protobuf have more of a learning curve.

Perhaps there are options that use what we currently have though? For example, picking up a compressed archive of RESO Common Format files using a pre-signed URL might be practical for initialization outside of a "reasonable" window.

Regarding historical data, RESO has an existing format for change logs and Data Dictionary entities called HistoryTransactional. With the DD 2.0 version, this also includes the EntityEventSequence so that the "pulse" that occurred can be associated with a transaction record.