Building your own video devices

Editor's Notes

• #2: Hello My name is Olga and I’m a Sr. Software Engineer at CrowdOptic. At CrowdOptic, we work with cutting edge technology for Smart Glasses. And we do live streaming and have patents for analyzing sensors. Previously, I worked at Salesforce and so know all about the difficulties working with enterprises. In this talk, I’ll describe how we built our own streaming device, what we have achieved so far, and the challenges we faced in the update process as well as in the delivery of the first version to a customer
• #3: In this era of 3-D printing and lego electronic blocks, it is now easier than ever to build a hardware prototype. At CrowdOptic, we had the idea to build our own streaming device for some time. So, when a couple of customers asked about a static device additionally to a Glass, it was our moment. And we built it. I’ll show you a demo of our product first. Here is our web client from which we can view, store, administer, and report on all of our devices. We can start streaming from here – start it. We have 2-way audio – from the device and to the device. After I stop it the video will be generated and available under archives. Click it and show the video. We have customers in medicine, security, government and other areas. Medical recordings and live streaming of highly technical surgeries will become an essential tool in medical training. Second - we have another application for our patented technology. I’m starting to stream from our CrowdOptic Eye device and Google Glass. Each of these has sensors to identify gps location and heading. When 2 devices view the same region, their line of sights intersect. We call this a cluster. The clusters identify if several people are looking at the same region. For now, we are applying this technology to some military customers. We took a Raspberry PI, added its camera part, wifi plug, power and a switch. And there we have a fully functioning hardware device.  We already had a client app for the Google Glass. It didn’t take too much time to repeat the same communication for live-streaming to our server. Soon after this, I realized that it is hard to track what the device is doing. When it connects or not to our server and when it streams. I quickly came up with the idea to add LED to show this. Pause We capture the video from the camera using a command line utility  - raspivid. Combining this with a powerful open source product -  ffmpeg we ended up with our first prototype
• #4: This is the main line that sends a camera stream to ffmpeg We use stdbuf to pipe a raspivid stream to ffmpeg as input Output for ffmpeg is our server address. Securing a stream is straightforward. SERVER_ADDRESS is localhost where a proxy is running to encrypt the data and send to our server remotely. That’s it. Our own streaming device!
• #6: The sales team wants to sell something even before it is built. We had a lot of pressure to finish devices that were already bought by a customer. When a device is already sold the worst thing is blindness. So first rule is…
• #7: Testing is not a QA only job. It is collaborative work between developers and testers. Whoever designs the product knows best how to simulate different conditions. It is time well spent to design the architecture in such a way that it will be easy to test. When we shipped our first version, we had some bugs. Sending logs using aws log agent and showing the status using LED helped to fix those quickly. For manipulating the device I decided to use an API. It allows us to extend to Smart Glasses and any device types we might have. For example we can use an API to start and stop streaming, which we extensively use in our testing. We also can use an API to modify device properties.
• #8: When something like this happens nobody is happy. Here’s where rule number 2 comes in
• #9: If the audio crashes, the video still needs to run The code that analyzes video quality and calculates several parameters should be highly guarded with try catch and be isolated if something unpredictable happens. Make sure to walk through the whole flow and identify actions that are not critical. pause It is common now to release a product where certain features can be disabled before the official rollout happens. Rule3
• #10: For example enableWebRtc when false should not execute any code(server side, ui) related to webrtc. This will give you an option to disable certain features if they are not working in certain customer conditions or not allowed. For example video file encryption in one of our China servers Additionally every customer is different and requires different settings. We developed eyeconf ui(show it) for configuration properties. It also shows available networks and which one is connected Rule 4:
• #11: It helps to play with timeout values for certain networks. Or specify bitrate, quality and frames per second especially for demo when using a weak connection. Pause When upgrading a product, the production environment can be different than testing. It is not always possible to keep up with the pace of change and all the customer’s software/hardware differences. This brings us to rule number 5.
• #12: It took us a month to clean up an update we released on all devices for a customer. We didn’t expect that kind of failure. It was related to a kernel problem because of OpenCv and some other library. It took 5-10 minutes to upgrade. And then a month to find and fix the problem. pause All enterprise customers have their own infrastructure - connection channel bandwidth and firewall rules. Don’t assume anything. Rule 6 says
• #13: Chasing why our livestreaming didn’t work in IE10 only at a customer place was an almost impossible exercise. There is no remote access to their network to test. And there are limited troubleshooting sessions which can be scheduled. Make sure all the ports are configurable and you can easily change them by a script. This way it will be possible to run some tests at the customer’s site. I have 2 security rules that were the most important for us:
• #14: It seems that it would be easier to place something to troubleshoot the product. For example if a certain password is supplied, it could show much more information about the state and allow it to stream or execute certain apis. Don’t do that and prohibit others from doing that. Remind everyone of that!
• #15: Completely no backdoors in the hardware. Consider rule number1: Always think about testing and tools to fix a problem immediately.
• #16: Pause The next rule says
• #17: If you think adding key2= random number along with another existing key= random number parameter in your api will improve your security, don’t be fooled. To improve security, verification data has to be gathered from different places, by different channels and can be modified in case of a breach. Now we come to an interesting rule number 9
• #18: We use ffmpeg, spring, Angular, bootstrap and many more. For our SIP implementation we started with command line utility – sippie. That’s a good start to combine all ready to use solutions. Only after that, revise to see what you have and if you really need that third-party software or instead if you can focus on developing your own. Think twice if you use a library for only 2-3 methods. It can be beneficial to implement those and avoid that library completely. pause At CrowdOptic we solve challenging and unique problems. You can’t find many solutions at StackOverflow. This time I decided to share my knowledge by posting answers for the problems I solved. I use stackoverflow, ffmpeg or Raspberry PI groups, linkedin connections(yes people are answering there highly technical questions), meetups. And ready to use code in Open Source projects. The last rule number 10 says:
• #19: Valuable information is around us, make sure you know how to use search efficiently. And not afraid to ask people for a help.