Data feed V1 EOL (end of life) support
Data feed V1 will no longer be supported as of January 31st. V1 data feed exports will be completely shut down.
We are migrating our underlying data feed database to Snowflake, this will enable you to have access to following to features:
- Access near realtime crawled data directly in a Snowflake database shared with your account (updates daily are made in snowflake daily)
- Additional export destinations, we now support sending data feeds to the following destinations:
- Amazon S3
- Google Cloud Storage
- Azure Blob
- Customizable export job schedules, we can now define any schedule to export your Forager data feed to your destination. Have data sent to your export destination at any frequency (charges do apply).
- Additional file formats, we now can support the following:
- JSON (NDJSON format)
- CSV/TSV
- Parquet
- Compression can now be applied to your data feed export files using any of the following:
- Gzip (default)
- bzip2
- Brotli
- Zstandard
- Ability to opt into net new data only. Only export data that has been updated from last export.
View the full OpenAPI schema documentation here.
View new file path and name formats here
- File export compression, all files are now exported using gzip by default, you do have the option to disable this or select from one of the other supported compression types above.
- JSON export format, all JSON files will now use NDJSON (one JSON object per line), instead of an array of JSON objects. See an example file here.
- File paths are now restructured to allow schema changes and to easily retrieve updated data only. For example:
/forager_data_feed/<data_feed_id>/<date_created>/<data_feed_type>_<schema_version>/data_<partition_id>.<file_type>.<compression_type>. See changes here. - Properties no longer supported:
background_pictureis_creatoris_influencertemporary_statustemporary_emoji_statusaccessibility_hashtagsaccessibility_textaddressprimary_localesupported_locales