Skip to main content

Documentation Index

Fetch the complete documentation index at: https://x-preview-mintlify-a8a065c0.mintlify.app/llms.txt

Use this file to discover all available pages before exploring further.

This endpoint has been updated to include Post edit metadata. Learn more about these metadata on the “Edit Posts” fundamentals page. 

​
Decahose stream

Enterprise This is an enterprise API available within our managed access levels only. To use this API, you must first set up an account with our enterprise sales team. Learn more The Decahose delivers a 10% random sample of the realtime X Firehose through a streaming connection. This is accomplished via a realtime sampling algorithm which randomly selects the data, while still allowing for the expected low-latency delivery of data as it is sent through the firehose by X. Below are some of the features available with Decahose:
  • Expanded and enhanced URLs: - fully unwinds shortened URLs and provides additional metadata (page title and description)
  • Stream partitioning - 2 partitions, each containing 50% of volume of the Decahose stream
  • Enhanced reliability - geographic diversity of backend systems
Note: This data is delivered in bulk, and does not support additional filtering (e.g. for keywords). ENTERPRISE

​
Streaming likes

This is an enterprise API available within our managed access levels only. To use this API, you must first set up an account with our enterprise sales team. Learn more Likes enable insight into who likes Posts and delivers accurate counts of likes. Gnip’s Firehose and Decahose can deliver public likes related to the Posts delivered via Gnip. This yields real-time public engagement and audience metrics associated with a Post.   Getting started with Likes As you prepare to consume likes data, you should know that:
  • Likes are delivered via an independent, separate stream
  • Likes are historically referred to as “Favorites”. The enriched native format payload maintains this nomenclature
  • Streams include only public likes
    • Public means that the liking user, Post creator and Post are all public on the platform
  • Likes are very similar to Retweets and represent a public signal of engagement
  • Payload elements include:
    • Original Post object
    • Actor object that created the original Post
    • Actor object that performed the like action
  • Only original content can be liked
    • Retweets cannot be liked. A like of a Retweet is applied to the original Post
    • Quoted Tweets can be liked
  • Like activities include applicable Gnip Enrichments (where purchased/applied)
  • Supported Products / Features
    • Likes streams support Backfill (where purchased/applied)
    • There is no Replay support for likes streams
    • There is no Search or Historical support for likes
    • There are no immediate plans to add likes support to PowerTrack
Decahose Native enriched format payload 
{
   "id":"43560406e0ad9f68374445f5f30c33fc",
   "created_at":"Thu Dec 01 22:27:39 +0000 2016",
   "timestamp_ms":1480631259353,
   "favorited_status":{
      "created_at":"Thu Dec 01 22:27:16 +0000 2016",
      "id":804451830033948672,
      "id_str":"804451830033948672",
      "text":"@kafammheppduman",
      "source":"\u003ca href=\"http:\/\/x.com\/download\/android\" rel=\"nofollow\"\u003eX for Android\u003c\/a\u003e",
      "truncated":false,
      "in_reply_to_status_id":803694205163814912,
      "in_reply_to_status_id_str":"803694205163814912",
      "in_reply_to_user_id":2855759795,
      "in_reply_to_user_id_str":"2855759795",
      "in_reply_to_screen_name":"kafammheppduman",
      "user":{
         "id":2855759795,
         "id_str":"2855759795",
         "name":"delirdim kanka",
         "screen_name":"kafammheppduman",
         "location":"sanane",
         "url":"http:\/\/instagram.com\/kafammheppduman",
         "description":"Manit @GalatasaraySk \ud83d\udc9e",
         "translator_type":"none",
         "protected":false,
         "verified":false,
         "followers_count":3702,
         "friends_count":607,
         "listed_count":1,
         "favourites_count":113338,
         "statuses_count":389,
         "created_at":"Sat Nov 01 22:38:25 +0000 2014",
         "utc_offset":null,
         "time_zone":null,
         "geo_enabled":true,
         "lang":"tr",
         "contributors_enabled":false,
         "is_translator":false,
         "profile_background_color":"C0DEED",
         "profile_background_image_url":"",
         "profile_background_image_url_https":"",
         "profile_background_tile":false,
         "profile_link_color":"1DA1F2",
         "profile_sidebar_border_color":"C0DEED",
         "profile_sidebar_fill_color":"DDEEF6",
         "profile_text_color":"333333",
         "profile_use_background_image":true,
       "Profile_image_url": "http:\/\/pbs.twimg.com\/profile_images\/804421763945861121\/v3bp9pnq_normal.jpg",
         "Profile_image_url_https": "https:\/\/pbs.twimg.com\/profile_images\/804421763945861121\/v3bp9pnq_normal.jpg",
         "profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/2855759795\/1480630085",
         "default_profile":true,
         "default_profile_image":false,
         "following":null,
         "follow_request_sent":null,
         "notifications":null
      },
      "geo":null,
      "coordinates":null,
      "place":null,
      "contributors":null,
      "is_quote_status":false,
      "retweet_count":0,
      "favorite_count":0,
      "entities":{
         "hashtags":[],
         "urls":[],
         "user_mentions":[
            {
               "screen_name":"kafammheppduman",
               "name":"delirdim kanka",
               "id":2855759795,
               "id_str":"2855759795",
               "indices":[
                  0,
                  16
               ]
            }
         ],
         "symbols":[]
      },
      "favorited":false,
      "retweeted":false,
      "filter_level":"low",
      "lang":"und"
   },
   "user":{
      "id":774146932365070336,
      "id_str":"774146932365070336",
      "name":"Uyuyan Adam",
      "screen_name":"saykoMenn",
      "location":"Tarsus, T\u00fcrkiye",
      "url":"http:\/\/connected2.me\/pmc1i",
      "description":null,
      "translator_type":"none",
      "protected":false,
      "verified":false,
      "followers_count":414,
      "friends_count":393,
      "listed_count":0,
      "favourites_count":9868,
      "statuses_count":370,
      "created_at":"Fri Sep 09 07:26:26 +0000 2016",
      "utc_offset":null,
      "time_zone":null,
      "geo_enabled":false,
      "lang":"tr",
      "contributors_enabled":false,
      "is_translator":false,
      "profile_background_color":"F5F8FA",
      "profile_background_image_url":"",
      "profile_background_image_url_https":"",
      "profile_background_tile":false,
      "profile_link_color":"1DA1F2",
      "profile_sidebar_border_color":"C0DEED",
      "profile_sidebar_fill_color":"DDEEF6",
      "profile_text_color":"333333",
      "profile_use_background_image":true,
      "Profile_image_url": "http:\/\/pbs.twimg.com\/profile_images\/802992813424201728\/VMzcTL3x_normal.jpg",
      "Profile_image_url_https": "https:\/\/pbs.twimg.com\/profile_images\/802992813424201728\/VMzcTL3x_normal.jpg",
      "profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/774146932365070336\/1480283382",
      "default_profile":true,
      "default_profile_image":false,
      "following":null,
      "follow_request_sent":null,
      "notifications":null
   }
}
Like Delete / “Unlike” payload 
{
   "delete":{
      "favorite":{
         "tweet_id":696615514970279937,
         "tweet_id_str":"696615514970279937",
         "user_id":2510287578,
         "user_id_str":"2510287578"
      },
      "timestamp_ms":"1480437031205"
   }
}

​
Guides

​
Recovery and Redundency

Introduction  With streaming high volumes of realtime Posts comes a set of best practices that promote both data reliability and data full-fidelity. When consuming realtime data, maximizing your connection time is a fundamental goal. When disconnects occur, it is important to automatically detect that and reconnect. After reconnecting it’s important to assess if there are any periods to backfill data for. The component that manages these details and consumes realtime Posts is only one part of a system with network, datastore, server, and storage concerns. Given the complexity of these systems, another best practice is to have different streaming environments, with at least separate streams for development/testing and production. Decahose comes with a set of features that help with these efforts.
  1. To support multiple environments, we can deploy Additional Streams for your account. These streams are independent of each other and have a different stream_label  to help differentiate them.
  2. To help support maintaining a connection, each Decahose stream supports Redundant Connections. The most common architecture is for a stream to have two connections, and on the client-side there are two independent consumers –ideally on different networks. With this design, there can be redundancy across the client-side networks, servers, and datastore pathways. Note that a full-copy of the data is served on each connection and the client-side must be tolerant of and manage duplicate data.
  3. A ‘heartbeat’ will be provided every 10 seconds; however, with the Decahose stream, the volume of data is high enough that even a small duration (e.g., a few seconds) of no Posts can indicate a connection issue. Therfore, both a ‘data silence’ and lack of a heartbeat can be used to detect a disconnect.
Since disconnects will happen, the Decahose stream has a dedicated Recovery and a Backfill feature to help recover data that was missed due to disconnections and other operational issues.

​
Additional Streams

Having additional Decahose streams is another way to help build reliability into your solution. Any additional streams are completely independent, having their unique endpoint. Each stream is assigned its own stream_label, and this label, along with your account name, are part of that stream’s URL. See the example below: https://gnip-stream.x.com/stream/sample10/accounts/:account\_name/publishers/twitter/:stream\_label.json The most common convention is to have a realtime stream dedicated for you production system, and an additional stream available for development and testing. Having a test/development stream enables Decahose customers to have a stream to test client consumer updates. While any (unique) label can be assigned to a stream, one convention is to use ‘prod’ for production stream, and ‘dev’ or ‘sandbox’ for an additional development stream. The number of streams, and their unique labels, is configurable by your account representative. Redundant Connections A redundant connection simply allows you to establish more than one simultaneous connection to the data stream. This provides redundancy by allowing you to connect to the same stream with two separate consumers, receiving the same data through both connections. Thus, your app has a hot failover for various situations, e.g. where one stream is disconnecte