In Part 1 of this series, Ray Tsang introduced our interactive demo Cloud Spin. Our goal was to capture and render a 180-degree, frozen-in-time, video using mobile technology and Google Cloud Platform. Why present a slide with a bunch of bullet points when you can demonstrate the power of Cloud Platform with wacky jumps in the air? In this post, I'll show how we captured the raw video using Android phones as cameras and Firebase to coordinate the devices.
We put the demo together with as many off-the-shelf components as possible. Our hope was that one day a few friends could get together, pull down the Cloud Spin source code from GitHub, pool their Android phones together, and make their own frozen-in-time videos. Note: the code isn't currently on GitHub, so watch the Cloud Spin repository for updates.
This means we were scrappy. REALLY scrappy. Our first prototype used three Nexus 6 phones held up by selfie sticks. The sticks were so unstable, that rolling by in an office chair would cause them to topple over. I repeatedly had to dive out of my chair to catch a phone before it smashed to the ground.
We used the Android camera2 API to automate recording the video. I started from the android-Camera2Video sample project on GitHub, which demonstrates how to record video with the camera2 API, save the captured video and display a live preview on the phone screen.
We used Firebase to handle this communication because it can be set up quickly and acts as a database that automatically sends real-time updates to mobile and web apps. Using Firebase, we connected the recording app on the phones to the control app running on a tablet, in real-time, without writing any back-end code.
For a sense of how little code it takes to use Firebase, here's a Java code snippet from the recording app that shows how the app reports the current status of the phone.
That's all the code needed to update the phone status. Within milliseconds, Firebase automatically pushes out the updated status to all mobile or web apps that have registered a listener.
The following screenshot of the Firebase web dashboard shows the data structure Firebase uses to store the phone status.
The following shows the data structure Firebase uses to store the snaprequest time.
The control app monitors the status of each phone and the version of the recording app. Each phone needed to use an external microphone so we could pipe the audio cue in through the headset port. This ensures that the audio track of the video is silent except for the beep sent from our control app running on a tablet.
The following snippet from the recording app shows the code used to upload the videos to Cloud Storage.
Note: This sample uses hard-coded credentials, which you would not want to do in a production version of this app.
To capture the video, we created four Android apps:
For the demo at Google Cloud Platform Next, the humble hardware prototype described in this post evolved into a custom-made, semi-circular stand with permanent phone mounts so we didn't have to guess at tripod placement, and a recording space big enough for me to pretend I had breakdancing skills.
That's how we turned videos from 19 different phones into an animated GIF of a single point in time.
-Posted by Bret McGowen, Google Cloud Platform Developer Advocate
 |
| Look at that hang time! |
This means we were scrappy. REALLY scrappy. Our first prototype used three Nexus 6 phones held up by selfie sticks. The sticks were so unstable, that rolling by in an office chair would cause them to topple over. I repeatedly had to dive out of my chair to catch a phone before it smashed to the ground.
 |
| The least stable way to hold up a half-pound phone |
Capturing a single moment in time
The first step was to figure out how to make each phone take a photo at the exact same instant. As mentioned in Part 1, we solved the problem by recording an 8-second video with each phone and marking the capture moment with an audio beep.We used the Android camera2 API to automate recording the video. I started from the android-Camera2Video sample project on GitHub, which demonstrates how to record video with the camera2 API, save the captured video and display a live preview on the phone screen.
 |
| Camera app reporting green (ready) as well as a live preview |
Aligning the cameras
The live preview was invaluable when we aligned the cameras. Proper alignment ensures the subject is in the same location from frame to frame, resulting in a smoother final video. To keep things simple, I displayed an overlay of two crosshairs on the preview screen of each phone; we marked two crosses under a tripod positioned where the person would jump and then moved each phone until the crosshairs lined up with the marks on the pole. Who says we can't be low-tech?
Communicating real-time status
Our next task was communicating between the phones and a control app. Each phone needed to receive one crucial piece of data as quickly as possible: the time to start recording video. The phones also needed to send out two values: a status (online, offline, error, uploading) and its relative position in the semicircle.We used Firebase to handle this communication because it can be set up quickly and acts as a database that automatically sends real-time updates to mobile and web apps. Using Firebase, we connected the recording app on the phones to the control app running on a tablet, in real-time, without writing any back-end code.
For a sense of how little code it takes to use Firebase, here's a Java code snippet from the recording app that shows how the app reports the current status of the phone.
// Set this phone's status on Firebase at /cameras/{cameranumber}/
mFirebaseRef = new Firebase(FIREBASE_BASE_URL);
mCameraStatusRef = mFirebaseRef.child("cameras").child(cameraNumber);
CameraStatus mCameraStatus = new CameraStatus();
mCameraStatus.appVersion = BuildConfig.VERSION_NAME;
mCameraStatus.externalMic = mExternalMic;
...
mCameraStatus.status = status;
mCameraStatusRef.setValue(mCameraStatus);That's all the code needed to update the phone status. Within milliseconds, Firebase automatically pushes out the updated status to all mobile or web apps that have registered a listener.
The following screenshot of the Firebase web dashboard shows the data structure Firebase uses to store the phone status.
Controlling the phones
To trigger a new video recording session, the control app updates the/snaprequest/timestamp value in Firebase. Firebase then immediately sends out the new value to the recording app on each phone. The recording app would then start the phone recording at the time specified by the timestamp.The following shows the data structure Firebase uses to store the snaprequest time.
Uploading video
After the phones recorded simultaneous video, there was one final task: uploading the video to Google Cloud Storage. Cloud Storage provides a nifty feature: Object Change Notification. If you create a notification channel for a storage bucket, and subscribe an application as a listener, Cloud Storage will send out a notification to all listeners whenever objects in the bucket change. We used this feature to automatically kick off the back-end video processing application when the recording app uploaded video from a new Cloud Spin session. (The video processing app is described in Part 3 of this series.)The following snippet from the recording app shows the code used to upload the videos to Cloud Storage.
Note: This sample uses hard-coded credentials, which you would not want to do in a production version of this app.
// Name of the Google Cloud Storage bucket.
String bucket = "my-cloudspin-bucket";
// Destination URL in the Cloud Storage bucket.
String cloudStoragePath = "20150821-151824-892/videos/01.mp4";
// Video file to upload.
File videoFile = new File("...");
// Credentials for Google Cloud Platform.
Credential credential = ...
Storage storage = new Storage.Builder(new NetHttpTransport(),
new JacksonFactory(),
credential).setApplicationName("...@developer.gserviceaccount.com")
.build();
StorageObject storageObject = new StorageObject();
storageObject.setBucket(bucket);
InputStream fileUploadStream = new FileInputStream(videoFile);
try {
InputStreamContent content = new InputStreamContent("video/mp4", fileUploadStream);
Storage.Objects.Insert insert = storage.objects().insert(bucket, null, content);
insert.setName(cloudStoragePath);
insert.execute();
} finally {
fileUploadStream.close();
}
Building the hardware prototype
Now we were ready test our design. We ordered 19 Android phones (one for every 10 degrees in a 180-degree arc, plus the starting point), ditched the selfie sticks for tripods with phone mounts, and even bought a few lights for our makeshift studio.
To capture the video, we created four Android apps:
- CameraApp – runs on the Nexus 6 phones to record the video and upload it to Google Cloud Storage.
- ControlApp – runs on a tablet to show the status of the phones running CameraApp, for debugging. When you click Take Photo, ControlApp sends an audio beep through the headphone jack, where it was split 19 ways, and into the headset jack of each Nexus 6 phone.
- RegistrationApp runs on a tablet to collect the name and contact information of the demo participant so we can send them their awe-inspiring work of gravity-defying art via email or Twitter.
- ApprovalApp showed the demo participant their final rendered video for their approval. If they had an awkward jump or inspiration for a new pose struck them while mid-air, they could reject and retake the photo.
Up next
Our demo was a success! Well, so far. For each session, Cloud Spin generated 19 separate video files and uploaded them to Google Cloud Storage. Now we needed to turn those separate video files into a single 180-degree animated GIF. In the next installment of this series, “Cloud Spin, Part 3: processing video using Google Cloud Platform,” Francesc Campoy will explain how we:- Extracted the frame corresponding to the audio cue and stitched the frames together into an animated GIF using Google Compute Engine.
- Parallelized the video processing with clusters of virtual machines to speed up rendering the final animation and scale to meet high demand.
- Orchestrated the parallel processing with Google Cloud Pub/Sub so that each video was processed, and only by a single server in the cluster.
That's how we turned videos from 19 different phones into an animated GIF of a single point in time.
-Posted by Bret McGowen, Google Cloud Platform Developer Advocate
