Daniel Ng
Email Me Contact Me

You're welcome to email me at dng11@sympatico.ca.

While I do like hearing from you, I am not interested in SEO services nor offshoring my business, so please don't offer me any.

Follow me on Twitter Subscribe to my RSS feed

Tips & Tricks: Using LINQ-2-Twitter to Embed Tweets in .NET Application
Originally posted by Daniel Ng on July 21, 2013.

The quick and painless way to embed tweets on your website is to use Twitter widget. There are plenty out there. Some are free and some are not. In most instances, you will be obliged to use their designs. If you are ok with that, then this post is probably not meant for you. If you enjoy coding and like to play with your own custom design, then read on.

To help us interface with Twitter more easily, we will be leveraging LINQ-to-Twitter, which is a 3rd party open source .NET class library authored by Joe Mayo. This is a cool API to work with. Another option would be to use Twitterizer, which is also free.

In this post, I will share some of my tips and tricks with you regarding LINQ-to-Twitter and Twitter development in general.


Tip #1: Get a Twitter account & get your keys!

Let's start with the obvious. You will need a Twitter account. Create one if you don't already have one.

Tip: To create an application, click on the arrow key right beside your profile photo at the top of the page.

Next, navigate to the Twitter Developers site and log in using the same account. You will need to create an application. Once you have one created, you will be provided with a Consumer Key and a Consumer Secret.

If all you need to do is basic searches for public tweets, then having these 2 keys would suffice. We will use them to create an application-only authentication to Twitter. But if you intend to interact with Twitter in a more meaningful way (e.g., sign into Twitter under a particular user account and create a new tweet), you will need to obtain an Access Token and an Access Token Secret as well. This is beyond the scope of this post.


Tip #2: Download LINQ-to-Twitter DLL

You may download the class library from http://linqtotwitter.codeplex.com/. At the point of writing this, the most recent version is v2.1.07. The downloaded zip file will contain a number of DLLs. Make sure you use the right one according to your .NET framework version.

Extract the proper DLL and add it to your Visual Studio project. Consider importing the LinqToTwitter namespace into your class.


Tip #3: Understand Twitter API rate limits

Tip: To find out the exact number of credits, check out this link https://dev.twitter.com/docs/rate-limiting/1.1/limits.

The Twitter rate limits have changed in v1.1 of the API. In summary, this is how it works:

Within a 15 minutes window, you have a total of X credits to spend. If you exhausted all your credits within that window, you will get a connection error. Your credits will be reset to the full amount every 15 minutes. The number of credits you have will depend on the authentication method you use (i.e., user vs application) and the API method call you make.

For instance, when using application-only authentication (i.e., connecting to Twitter under the identity of an application and not the identity of any user in particular) to look up public tweets by Screen Name, you will have 300 credits to spend within a 15 minutes window. Each search will cost you one credit. Let say you want to retrieve tweets from 50 different users (i.e., 50 searches), you are only allow to execute your 50 searches a maximum of 6 times within that window (i.e., 300 credits / 50 credits = 6).

The main point here is that you really need to manage how many and how often you do your searches or you may risk getting your account suspended by Twitter if you often go overboard your limit.


Tip #4: VB code sample

Here is a VB code sample for retrieving and displaying tweets. Only a day-old public tweets will be returned from the query. Notice that I intentionally exclude all replies using the ExcludeReplies() property. You may want to do that in some cases. Remember to plug in your own Consumer Key and Consumer Secret. Make sure you import the LinqToTwitter namespace as well.

'*** Type in your own consumer key and consumer secret here. ***
Dim oCredential As New InMemoryCredentials
oCredential.ConsumerKey = "Enter your consumer key here"
oCredential.ConsumerSecret = "Enter your consumer secret here"

'*** Use application-only authentication to connect to Twitter. ***
Dim oAuthorizer As New ApplicationOnlyAuthorizer
oAuthorizer.Credentials = oCredential
oAuthorizer.Authorize()

'*** Define a list of screen names to check for public tweets. ***
Dim screenNames As String = "LambtonAlerts,LambtonFireInfo,SarniaPolice"

'*** Define a cutoff date, e.g., 1-day old. ***
Dim cutoffDate As Date = Now.Subtract(New TimeSpan(1, 0, 0, 0))

'*** This is to remove the time from the date. ***
cutoffDate = cutoffDate.Date

Using oTwitterContext As New TwitterContext(oAuthorizer)

  '*** Instantiate a master list of tweets. ***
  Dim allTweets As New List(Of LinqToTwitter.Status)

  '*** Need to check one screen name at a time. ***
  For Each s As String In screenNames.Split(",")

    Dim screenName As String = s.Trim

    If String.IsNullOrEmpty(screenName) = False Then

      '*** Get a list of 1 day old public tweets for a given user. ***
      '*** Exclude all replies. ***
      Dim tweets = (From tweet In oTwitterContext.Status _
                    Where tweet.ScreenName = screenName _
                          And tweet.Type = StatusType.User _
                          And tweet.CreatedAt.ToLocalTime >= cutoffDate _
                          And tweet.ExcludeReplies = True _
                    Select tweet).ToList

      '*** Add tweets to the master list. ***
      If tweets.Count > 0 Then allTweets.AddRange(tweets)

    End If

  Next

  '*** Display all the tweets from newest to oldest. ***
  For Each tweet In allTweets.OrderByDescending(Function(a) a.CreatedAt)
    Page.Response.Write(String.Format("{0}<br />[{1}] {2}<br/><br/>", _
                                      tweet.CreatedAt.ToLocalTime.ToString, _
                                      tweet.ScreenName, _
                                      tweet.Text))
  Next

  '*** To display current rate limit info. ***
  Dim resetWhen As New DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)
  resetWhen = resetWhen.AddSeconds(oTwitterContext.RateLimitReset).ToLocalTime()
  Page.Response.Write(String.Format("Remaining = {0}; Limit = {1}; Next Reset = {2}", _
                                    oTwitterContext.RateLimitRemaining, _
                                    oTwitterContext.RateLimitCurrent, _
                                    resetWhen.ToString))

End Using

Tip #5: Searching by Screen Name or User ID?

A Twitter's Screen Name can be changed by the account holder on a whim. Therefore, searching for tweets by Screen Name is not as reliable as searching by User ID. To find someone's ID from their Screen Name, visit this site http://mytwitterid.com/.


Tip #6: Linkify tweet message

Various entities in a Tweet

The tweet messages we get back from Twitter API will have all the hyperlinks removed. You are expected to write some custom code to linkify the message yourself.

Essentially, you have to loop through all the embedded entities, look up the original link, and then do a search-and-replace operation on the tweet message. The sample code below shows you how I would do this. But you should definitely check out this blog post from Nansen. It has an alternative way of handling this that is more efficient.

First, set up a basic structure like this in your class:

Structure ToReplace
  Public StartPosition As Integer
  Public EndPosition As Integer
  Public SearchString As String
  Public ReplaceString As String
End Structure

Next, while iterating through all your tweets, you would perform the following:

'*** Assign the original message to a variable. ***
Dim tweetMsg As String = tweet.Text

'*** Instantiate a new running list to store all entities. ***
Dim eList As New List(Of ToReplace)

Dim s1, s2, s3 As String 
s1 = "<a href='https://twitter.com/search?q=%23{0}'>#{0}</a>"
s2 = "<a href='{0}'>{1}</a>"
s3 = "<a href='https://twitter.com/{0}'>@{0}</a>"

'*** Add all hastag entities to a running list. ***
For Each entity In tweet.Entities.HashTagEntities
  eList.Add(New ToReplace With {.StartPosition = entity.Start, _
                                .EndPosition = entity.End, _
                                .SearchString = entity.Tag, _
                                .ReplaceString = String.Format(s1, _
                                                               entity.Tag)})
Next

'*** Add all URL entities to a running list. ***
For Each entity In tweet.Entities.UrlEntities
  eList.Add(New ToReplace With {.StartPosition = entity.Start, _
                                .EndPosition = entity.End, _
                                .SearchString = entity.Url, _
                                .ReplaceString = String.Format(s2, _
                                                               entity.Url, _
                                                               entity.DisplayUrl)})
Next

'*** Add all user mention entities to a running list. ***
For Each entity In tweet.Entities.UserMentionEntities
  eList.Add(New ToReplace With {.StartPosition = entity.Start, _
                                .EndPosition = entity.End, _
                                .SearchString = entity.ScreenName, _
                                .ReplaceString = String.Format(s3, _
                                                               entity.ScreenName)})
Next

'*** Add all media entities to a running list. ***
For Each entity In tweet.Entities.MediaEntities
  eList.Add(New ToReplace With {.StartPosition = entity.Start, _
                                .EndPosition = entity.End, _
                                .SearchString = entity.Url, _
                                .ReplaceString = String.Format(s2, _
                                                               entity.Url, _
                                                               entity.DisplayUrl)})
Next

'*** Need to do string replacement backward so as not to mess up the position index. ***
For Each item In eList.OrderByDescending(Function(a) a.EndPosition)
  tweetMsg = tweetMsg.Remove(item.StartPosition, item.EndPosition - item.StartPosition)
  tweetMsg = tweetMsg.Insert(item.StartPosition, item.ReplaceString)
Next

'*** Here is your new message. ***
Page.Response.Write(tweetMsg)


Tip #7: About Twitter datetime values

Twitter stores all datetime values in UTC (Coordinated Universal Time). This is not a big deal as long as you remember to convert them to local datetime before working with them.

The simplest way is to use the DateTime.ToLocalTime() method to do the conversion for you (see Tip #4). Alternatively, the LinqToTwitter.User entity has a UtcOffset() property which you can use to calculate local datetime.


Tip #8: Consider caching

If the situation permits and when it makes sense to, consider caching your tweets. In my particular case, I adopted the following strategy:

Caching tweets to an XML file

I have set up a dedicated process (i.e., a custom Windows Service) that will check Twitter for tweets at regular interval. All downloaded tweets are then written to a local XML file. I will then have other applications read that XML file instead of connecting to Twitter directly.

Some of the advantages are:
  1. Improve application performance.
    Since my ASP.NET application no longer connects to Twitter directly, its performance should improve. This would be more evident if I have a time-consuming query. Furthermore, I don't have to bother with writing code to fetch tweets asynchronously to mitigate performance issue.
  2. Have better control over the Twitter API rate limits and not exceeding it.
    I now have a separate process that connects to Twitter and this process will no longer be influenced by my user actions. How often that process connects to Twitter is also carefully regulated so as not to exceed the rate limits.

Tip #9: Re-direct to mobile.twitter.com?

For Internet Explorer 6 & 7 users, clicking on any hyperlink on the tweet message will redirect you to the mobile.twitter.com site instead of twitter.com. This is a known issue. And there is no easy fix other than to upgrade your IE or to use a different browser.

For Internet Explorer 8 & 9 users, if you are being redirected, you need to disable the compatibility view in your browser settings.


Final Notes

Feel free to email me if you have any questions or concerns. You may also want to consider subscribing to my RSS feeds for future posts.

Copyright © 2013 www.xcentricway.ca. All rights reserved.