Upstream Messages With Google Cloud Messaging

Google announced a new messaging model with Google Cloud Messaging on last years Google IO. This model allows for upstream messaging and for user specific notifications. This post is the first in a three-part series in which I am going to look into both these changes and to show you how to use them.

I had never written these posts if it weren’t for the Dutch Android User Group meetup in Utrecht (January 2014), where I had the pleasure to talk about this topic. This event was well-attended and was packed with interesting talks about the changes in Android 4.4 and Google Play Services. If you have ever the chance to attend a meetup of our Dutch colleagues, I strongly suggest to do so. It has been a gorgeous evening and was well worth the two-hour drive. It was not just the talks but also the overall atmosphere as well as the conversations during the breaks and afterwards. A big thanks to the organizers for the awesome evening!

You can find my slides on the speakerdeck site. And a video of the talk should also be available pretty soon.

Sample App

I have published sample apps on github where you can see how to use this in context. For the Android client clone the gcm_sample project. For the server clone the gcm_server project.

What’s new

With Google Play Services 3.1 Google made some changes to Google Cloud Messaging:

  • You can send messages upstream from the user’s device
  • You have to use a permanent connection between your backend and Google’s services
  • You can use user specific notifications

In this post I will only cover upstream messaging. That is the flow of events and what to do on Android’s side. In my next post I’m going to cover the server side – and in the third post I will write about user specific notifications. User notifications allow you to send user specific messages with the possibility to cancel notifications on other devices if they have been consumed already on one device.

Upstream messaging

With the new GCM release Google allows you to send messages directly back to Google’s cloud. That’s what they call upstream messaging. Have a look at the following diagrams. The first one is the traditional model of Google Cloud Messaging while the second one shows the new flow of events.

The classic communication flow with Google Cloud messaging

The classic communication flow with Google Cloud messaging


The communication model of Google Cloud Messaging with the new Cloud Connection Server

The communication model of Google Cloud Messaging with the new Cloud Connection Server

The first two steps in both diagrams are the same: You tell the Google Cloud Messaging API that you want to register a device (step 1). If the registration succeeds Google returns the registration id for this device (step 2).

Traditionally you would then send the registration id back to your server (step 3). The protocol was up to you but very likely HTTP. The server would (hopefully) respond with a success message (step 4).

From then on your server could send messages to Google’s Cloud Messaging service (step 5) and the user’s device would receive it, if it’s registration id was part of the registration_ids parameter of the message (step 6).

With upstream messaging though your app would send the registration id (step 3) not directly to your server, but instead call a method of the Google Play Services API. Google Play Services would then send the message to Google’s servers.

Google then processes the message and sends it to your backend. Obviously for this to work your backend must be connected in some way to Google’s cloud. I will deal with this part in my next post. For now it’s enough to know that there has to be a permanent connection between your backend and Google’s cloud and that Google will deliver messages that way (step 4).

After the registration id has been transmitted successfully you can send messages to the device (step 5). But with the new way to use Google Cloud Messaging you would use the existing connection between your server and Google’s servers and use a different protocol (with slightly different parameters).

Note that Google uses different names for its servers depending on which transmission model you want to use. Google uses the name “GCM HTTP Connection Server” for those servers receiving HTTP messages and it uses “GCM Cloud Connection Server” for those receiving XMPP messages via the permanent connection. The abbreviation CCS is very commonly used within GCM’s documentation.

Create your credentials

If you already use Google Cloud Messaging you can skip this part and proceed to the next section.

If you’re new to Google Cloud Messaging, you have to create an API key in Google’s Cloud Console/Google Developer Console. Do not confuse it with the Google Play Developer Console. Both are unrelated.

After logging in to the Cloud Console you will see a list of projects. Simply create a new one and give it a name and a project id in the next dialog. Now this could be a source of confusion. The project id you use here is not the project id you have to use in your code later on. It’s just a name which nevertheless has to be unique.

Afterwards creating the project the developer console will show you the project’s page. Go to “APIs & Auth” in the menu and turn on “Google Cloud Messaging for Android” in the huge list of Google’s services. Since Google displays all your active services at the top, the entry will move directly to the top after you enable it. Don’t be irritated by that tiny UX glitch. Simply scroll to the top and you will see that it’s status has changed to “On”. That’s great.

Switch to Credentials in the menu on the left. At the top of the new screen you can find a client id. It starts with a number followed by “.apps.googleusercontent.com”. Select the number (only the number), copy it and paste it into your strings.xml – or a special gcm_strings.xml that only contains the project id. Name the string whatever you like – in the sample project I expect a string named gcm_project_id.

That’s all you need for the Android part. But since you’re logged in to the dev console anyway, let me show you how to create the API key for your server right away.

Go back to the dev console. On the Credentials page there are two sections. The OAuth section on the top is where you have found the project id. Below of it is the section called “Public API access”. It’s still empty. Click “Create New Key” and in the next dialog select “Server key”. Afterwards you will see the API key for server applications together with some additional information. You will use the API key when following my post about the server side of GCM.

Sign up for using the new features

To be able to use all the shiny new features of Google Cloud Messaging, you have to sign up first. With the sign up form you “Sign up to be a trial partner”. The new services are not yet finalized. It’s a bit like it has been with Google Cloud to Device Messaging (C2DM) – the predecessor to GCM. You had to sign up as well, and when the service was finally made public, it had evolved to Google Cloud Messaging. So expect the API to change!

Using Google Play Services

You need to use Google Play Services from within your app for registering the device for as well as for sending upstream messages.

Setting up Google Play Services is easy, but requires too many steps to cover them in this post. And those steps also are different depending on whether you use Android Studio, Eclipse or something else. Have a look at the Play services documentation if it’s the first time you use Google Play Services.

The very first step in your code when using Google Play Services is to make sure that Google Play Services are available at all. You can do so with the following snippet:

private boolean checkPlayServices() {
   int resultCode = 
         GooglePlayServicesUtil
            .isGooglePlayServicesAvailable(getActivity());
   if (resultCode != ConnectionResult.SUCCESS) {
      if (GooglePlayServicesUtil.
            isUserRecoverableError(resultCode)) {
         GooglePlayServicesUtil.getErrorDialog(
               resultCode, 
               getActivity(),
               RC_RES_REQUEST).show();
      } else {
         Log.i("grokkingandroid", 
               "This device is not supported.");
      }
      return false;
   }
   return true;
}

What to do when the services are not available, depends on how important they are for your app. If Play Services are not essential, I suggest to let the user know – but in an unobtrusive way. After all the app is usable without it, so do not annoy your users with dialogs or other stuff that keeps them from doing what they actually installed the app for.

Registering the device for Google Cloud Messaging

The next step is to register the device. Google needs to know about your device before your app is allowed to receive or send messages.

In my sample app the user has to explicitly click the button register. Not a typical use case, but for the sample this makes sense. Now when to do this is in a real app is up to you. But when your app benefits from Google Cloud Messaging you usually want to register the device as soon as possible.

I also let the user select a Google account. Your app’s users do not need one for Google Cloud Messaging. I solely request one to use this account for user specific notifications. I will cover those in the third post of this series.

Registering actually is very easy. First get hold of a GoogleCloudMessaging object and then simply call its method register() passing in the project id of the dev console (see “Create your credentials” above):

String projectId = 
      getResources().getString(R.string.gcm_project_id);
GoogleCloudMessaging gcm = 
      GoogleCloudMessaging.getInstance(this);
String regid = gcm.register(projectId);

Be careful though: Asking for the registration id is a blocking request. Hence, you should always use the call in a separate thread off the main thread (UI thread). In the sample project I am using an IntentService to achieve this.

Send messages upstream

As soon as the device is registered, it can send messages upstream – as well as receive messages. Probably the very first message to send upstream is the newly received registration id.

Sending messages is nearly as simple as registering. You create a Bundle with all the payload data you want to transmit and you also have to create a unique message id. In my sample code I simply increase a counter I keep within the SharedPreferences for the message id. Depending on your payload the code might look more or less like this:

try {
   Bundle data = new Bundle();
   // the account is used for keeping 
   // track of user notifications
   data.putString("account", account);
   // the action is used to distinguish 
   // different message types on the server
   data.putString("action", Constants.ACTION_REGISTER);
   String msgId = Integer.toString(getNextMsgId());
   gcm.send(projectId + "@gcm.googleapis.com", msgId,
         Constants.GCM_DEFAULT_TTL, data);
} catch (IOException e) {
   Log.e("grokkingandroid",
         "IOException while sending registration id", e);
}

As you can see you once again need a GoogleClientMessaging object and call its send() message.

This message takes as parameters the project id appended with the fixed string “@gcm.googleapis.com“, a unique message id, a time to live value and the bundle with your payload.

You might notice that I do not include the registration id as part of the payload. That’s because Google Play Services adds the registration id to every message to your backend anyway. You will see in the next post which parameter Google uses for the registration id. For now, just keep in mind that above code is all that is actually needed.

Those strings within the bundle are not even necessary. I use the account because I want to send user specific messages and I use the action to differentiate on my server between different message types.

As you can see there is no special message for sending the registration id back home. It’s just a message like any other upstream message.

Benefits of Upstream messaging

Now that you know how to send upstream messages, you might wonder about why to use upstream messaging in the first place. Why not send your message back to your server as you always did (assuming you did contact your server)?

There are three reasons on the Android side for using the new model:

  • The upstream messaging API is dead simple
  • Upstream messages can reuse connections
  • Upstream messages are needed for notification cancellation

The API is dead simple

You have seen how simple the Android code is. It’s just some message calls to Google Play Services’s GoogleCloudMessaging class. It’s really just this one class, which with only six public methods is not too difficult to grasp :-)

Contrast that with using HTTP instead: When talking back to your own servers directly you have to take care of checking if the network is available, send off the message, check for any error conditions and enqueue the message for later delivery if anything goes awry. All in all you’re better of with upstream messaging – at least for as long as you only care about the Android code.

Upstream messages can reuse connections

Not only is the API easier to use, it can also save resources. The user’s device has a connection to Google anyway so with upstream messaging Google Play Services can reuse this connection. The services take care of using the most power efficient way to send messages so you do not have to deal with that. This helps keeping the battery usage of your app to a minimum.

Of course if your app only sends messages every once in a while, your users won’t notice any change in the battery usage. But if your app has to send many messages to your backend, this might make a difference to your users.

Upstream messages are needed for notification cancellation

The strongest argument for using upstream messaging though is notification cancellation. If you send messages from your server to specific users, you do not want them to see a notification on each and every device – even if it has long been read on another device. I will deal with user specific messages and notification cancellation in my third post about Google Cloud Messaging. So stay tuned :-)

Permissions

To use Google Cloud Messaging you need quite a lot of permissions:

<uses-permission 
   android:name="android.permission.INTERNET" />
<uses-permission 
   android:name="android.permission.GET_ACCOUNTS" />
<uses-permission 
   android:name="android.permission.WAKE_LOCK" />
<uses-permission 
   android:name="com.google.android.c2dm.permission.RECEIVE" />
<permission 
   android:name=
      "com.grokkingandroid.sampleapp.samples.notifications.gcm.permission.C2D_MESSAGE"
   android:protectionLevel="signature" />
<uses-permission 
   android:name=
      "com.grokkingandroid.sampleapp.samples.notifications.gcm.permission.C2D_MESSAGE" />

The first three are predefined permissions. You might notice the account permission. Actually this one is only needed for API levels before Android 4.0.4. I added it for completeness.

The fourth is a bit of an oddity. It’s the old Cloud to Device Messaging (C2DM) permission. For backwards compatibility it makes sense to not change it – otherwise existing apps had to ask for a new permission. But it looks weird about two years after C2DM was deprecated.

You also have to create a new permission of your own. That’s the permission that ensures that no other app can receive messages intended for your app. It has to exactly follow this naming convention: Package name plus “.permission.C2D_MESSAGE“. Otherwise your app won’t receive the messages sent to it. Your app also must use this newly created permission.

Receiving messages

Of course the fun part of Google Cloud Messaging is to actually receive push messages from the cloud. Google transmits the messages to your app to a BroadcastReceiver.

The following code is basically the same as Google’s sample code on their client page:

/**
 * See http://developer.android.com/google/gcm/client.html
 */
public class GcmBroadcastReceiver 
      extends WakefulBroadcastReceiver {
   @Override
   public void onReceive(Context context, Intent intent) {
      // Explicitly specify that
      // GcmIntentService will handle the intent.
      ComponentName comp = 
            new ComponentName(
               context.getPackageName(),
               GcmIntentService.class.getName());
      // Start the service, keeping the 
      // device awake while it is launching.
      startWakefulService(
            context, 
            (intent.setComponent(comp)));
      setResultCode(Activity.RESULT_OK);
   }
}

The Broadcastreceiver gets the message – but since it runs on the UI thread it should immediately pass the message on to some background thread. I use the same IntentService you have already seen for registering the device.

Actually parsing the message and doing what has to be done is done in the service. Again The code is pretty much the same as on Google’s documentation:

// handling of stuff as described on
// http://developer.android.com/google/gcm/client.html
try {
   Bundle extras = intent.getExtras();
   // The getMessageType() intent parameter
   // must be the intent you
   // received in your BroadcastReceiver.
   String messageType = gcm.getMessageType(intent);
   
   if (extras != null && !extras.isEmpty()) {
      /*
      * Filter messages based on message type. 
      * Since it is likely that GCM will be 
      * extended in the future with new 
      * message types, just ignore any message 
      * types you're not interested in, or that you
      * don't recognize.
      */
      if (GoogleCloudMessaging.MESSAGE_TYPE_SEND_ERROR
            .equals(messageType)) {
         sendNotification(
            "Send error: " + extras.toString());
      } 
      else if (GoogleCloudMessaging.MESSAGE_TYPE_DELETED
            .equals(messageType)) {
         sendNotification("Deleted messages on server: "
            + extras.toString());
      } else if (GoogleCloudMessaging.MESSAGE_TYPE_MESSAGE
            .equals(messageType)) {
         // Post notification of received message.
         String msg = extras.getString("message");
         if (TextUtils.isEmpty(msg)) {
            msg = "empty message";
         }
         sendNotification(msg);
      }
   }
} finally {
   // Release the wake lock provided by the WakefulBroadcastReceiver.
   GcmBroadcastReceiver.completeWakefulIntent(intent);
}

As you can see GCM uses different message types. the last one is the one actually containing a new messages. The first one informs you about errors when sending messages and the second tells you that messages have been deleted. If for example you display incoming messages with a notification you would have to cancel the notification in this case.

What to do when receiving an actual message is very specific to your app. The sample app simply displays a notification.

Do not forget to release the wake lock. That’s the line within the finally block. The finally block ensure that this call will always be done – even if any exception has been thrown in the code within the try block. Be sure to do it that way!

Lessons learned

In this post you have seen how the message flow has changed when using Google Cloud Messaging. With the previous model you had to contact your server directly from the device, now you only tell Google Play Services what to do with the new upstream messaging model. Google’s services take care of actually delivering the message to your server.

You also have learned about how to set up the project in Google’s developer console and that you have to sign in to use the new features.

Finally you have seen how easy the Android code actually is and what the benefits of upstream messages are.

In the next post you will learn more about the server side of upstream messaging – which is a tad more complex. And after that I’m going to show you how to use user notifications in the final post of this series.

Share this article:

You can leave a response, or trackback from your own site.

29 Responses to “Upstream Messages With Google Cloud Messaging”

  1. Mike V-C says:

    Hello Mr. Rittmeyer,

    This is a fantastic blog entry. My partner and I are trying to develop an app that would take advantage of this technology but trying to find solid examples beyond what Google on the development page is proving very difficult. We can’t thank you enough.

    I’m very new to xmpp and how to stand up an xmpp sever. I have experience setting up Tomcat but I’m not sure how to run your server example. Can you tell me what platform you are running your server on? You mentioned in the readme that you used a Java webapp, but I was wondering which one. Are there any good guides out there that would show me how to configure it?

    Thanks
    Mike V-C

  2. Muha says:

    Thanks for the great post! Looking forward to the next posts.

  3. Arjen says:

    Thanks for the great post. Will the upstream messaging still require the GCM ports to be opened in the firewall? This is my biggest issue with GCM, it uses non standard ports that are almost always closed on corporate firewalls.

    • The problem remains. It’s a connection to port 5235. I mentioned it briefly at my DAUG talk in Utrecht: For development it’s a big hurdle when testing your stuff. For example, while in Utrecht, I was not able to connect to Google’s servers via the local Wifi network. I think it’s less so a problem in production.

  4. SHady says:

    Hi Wolfram,

    I would like to thank you for the clear explanation for the GCM, I was searching for clear explanation with complete code for upstream as I was interested to send and messages between android and server and here I found it at your website, but I have a problem now that whenever I create an XMPP connection with SMACK I got project ##### not whitelisted, and I have filled the information in (https://services.google.com/fb/forms/gcm/) as I have read that it need to be activated through google and they replied that they will answer shortly but I read it might take up to 3 months, Sorry I appreciate if you tell me about another way of send messages from android to server without XMPP ?

    Thanks
    Shady

    • In my case it took them about three weeks, if I recall that correctly. I’ve heard that it lasts a bit longer now – but only Google knows about the queue of applicants and how fast they can process this queue.

      You could of course use the old way with HTTP, if you need GCM pretty soon. Since you can combine both modes, your work won’t be wasted if you later start to migrate to XMPP based messaging. I suggest to have a look at the official documentation as lots of blog posts use the deprecated GCM libraries instead of Google Play Services.

  5. Shady says:

    Thanks Wolfram, I went through the documentation in HTTP but I didn’t find a way to send from android to device or at least didn’t find a way for the server to listen to the messages coming from android device if I succeed to send messages to server using GCM.
    My main goal is to have chat between Website and android devices based on GCM.

    Shady

    • Well, there is no way. If you use HTTP and the common GCM way of handling things, than you must use HTTP to contact your backend directly from within your app. You cannot send messages upstream to Google’s cloud when not having been approved yet.

      It’s nevertheless no problem to realize a chat that way. You just have to take care yourself of checking for connectivity, queueing messages, reacting to responses and so on.

      • Shady says:

        Thanks wolfram, you are right I need to have like listener to handle incoming messages from android.
        Thanks again for clarifying things.

        Shady

  6. […] Google announced a new messaging model with Google Cloud Messaging on last years Google IO. This model allows for upstream messaging and for user specific notifications. This post is the first in a three-part series in which I am going to look into both these changes and to show you how to use them. I …  […]

  7. Kaustav Ghosh says:

    Dear sir,
    I am in deep trouble with an app and hope to get some help from you.I am developing a housing app in which there are 2 kinds of users:

    1.Complex owners.
    2.Flat Owners.

    Now there is an announcement section in which the complex owner will make an announcement(using the app only and not through the web) and all the flat owners will receive that announcement via push message.Something similar to Facebook push message(i.e. when someone comments or likes your status then a notification arrives at your android device).

    Now how can I achieve that??Is it possible to use Gcm?I am confused as gcm is used for server side messaging only.

    • Well, kind of. The message would have to go to your server first (for example via upstream messaging). Afterwards the server then can send it to all recipients via GCM (using either HTTP or XMPP).

      You can’t send a message to GCM recipients directly. But above solution should suffice for your use case.

  8. Stjepan says:

    Hi Wolfram,

    I’m not getting from your example where I can specify which users I wan’t to send the upstream message or this is not an option in this case?

    Thnx in advance

    • Upstream messaging is just another way to send messages to your server. Your server then might react to such a message in sending a Google Cloud Messaging message to specific registration ids or notification keys. The latter can only be used to target other devices of the same user – something which I’m going to cover in my upcoming post about User Notifications with upstream messaging.

  9. Luigi Manganiello says:

    Hello Mr. Rittmeyer,

    I followed the instructions of this post and I created a Test Android App and a 3rd party server using smack api 4.0.0
    With smack 4 I had to change some code lines (little changes in the smack API), but the server received the upstream message from device with null payload:

    null

    I downloaded the previous version of smack API (3.4.1) and I didn’t have to change any code line of your code and it worked fine!

    There is a bug in the new release of smack api? There is some sort of incompatibility?

    In any case I’d like to suggest to write the exact version of smack api used in your post to avoid other people will face the same issue.

    Thanks a lot for your post and thanks in advance if you want reply me on the issue.

    Best Regards.
    Luigi

    • Yes. You are right. The API changed and I should mention the version I have used for this example. I will add this information.

      If I recall it correctly I had tested an updated version (to Smack 4.0) as well, but am not sure about it. Can you provide a gist or a pastebin snippet with the relevant parts of your CcsClient code?

  10. Hi,

    i used GCM in few of my apps and now I want to implement CCS for a chat client where one can share message to all other users or to single user(broadcast or unicast messaging). Is it possible through CCS or not?

    in some place, its written that, with XMPP end-point CCS can send unto 4000 in one second and immediately on next line its written, multicast is not possible yet.

    I have seen ur example and its a quite good tutorial however something bothers me that is where Google will know my server address so that it will hand-over the message to it…You have no where mentioned that connection in your code.

    Please clarify my doubts.

  11. Another thing I wanted to share as well, that I have signed up for this API one week ago…suddenly I got an e-mail that one of the google’s guy will get back to me and no one is ever been in touch aid me since.

    My question is what will they do, will they open some connection for us or give us API key?

    • Since Google IO 2014 the approval process has been removed and all can immedietaly use all new features (some of which I haven’t covered yet). You get your API key in the dev console (Google developer console, not the Play developer console) – you actually should have one since you are using GCM already. Nothing else is needed to get GCM with CCS to work.

  12. Hi,

    I got ur point however two of things are still not clarified yet. First of them are possibility of broadcasting using CCS and second is where should I write upstream message code in my android client end where I will send message to my backend through GCm.(I mean I know its being fired using gcm.send() however I don’t know how it will know about whom to deliver cause I am planning to have millions of users in my database, is it the place where “Notification Key” will come to play a role). Please clarify some points about “User notification”.

    Its been my pleasure talking to such a great engineer through this post.

    • You cannot send bulk messages. But since XMPP is more efficient you should still be able to send more messages in the same time, even if you send to each registration_id individually. That’s at least what Google claims. I cannot confirm this since I do not send out millions of messages :-)

      But how to target users is not much different from the old model. Just the name of the attribute to use has changed. You now have to use “to” with one registration_id instead of “registration_ids” that you could use for an array of up to thousand registration_ids.

      All messages of your client contain the attribute “from” so that you know who sent it. Google Play Services adds that to the client message. You have to take care in your app, that you have registered the device prior to sending any messages.

  13. Okay, lets just integrate the whole thing into a straight-way representation and please rephrse me if I am going wrong.

    Question 1: – I got registration id through gcm.register and then I will have to send this to GCM or directly I have to send this to my backend?

    Question 2:- if I will send this to GCM then how will it reach my server and how my server will send downstream message?

    Question 3: – Scenario 2:- I send this to my server and my server will send (downstream message)new Model message through that “to” with one registration id at one time.

    Question 4: – then my device will receive that message and it will acknowledge the server with a upstream message through gcm.send(that persistent connection will look after by Google). My question here is what is the unique thing device will have to look for when answering upstream?

    Thanking you for your patience.

  14. I am not able to get the use of this method as well sendRegistrationIdToBackend(). In this method, you send the registration id through gcm.send(mSenderId + “@gcm.googleapis.com”, msgId,Constants.GCM_DEFAULT_TTL, data);
    how would it save the registration id in my database or server or its not required. Please clear up the things.

  15. I just downloaded the server side code and include the lib files(Smack and json) however I am finding error in CCSClient. So I read the thread again and replace the recent Smack with 3.4.1 however the same error exists.

    The error appears near “GCMPacketExtension” class’s

    public Packet toPacket() {
    return new Message() {
    @Override
    public String toXML() {

    as its not finding toXML. What to do and Another thing I wanted to ask you that after I register the device, who will initiate the message process, shall send message from device or from web I send. How will I send message from web? is there any interface?

  16. Mark cruz says:

    Great information. Your post is really helpful. You can see more here related to similar issue i.e. http://bit.ly/1pTtyh4

Leave a Reply

You can also subscribe without commenting.

Subscribe to RSS Feed My G+-Profile Follow me on Twitter!