Scarlett Application Redesign Part 2: Next Steps and Poof-of-Concept !

So a couple of weeks ago, I posted this blog post asking for some help on redesigning Scarlett to it easier to develop her/add more features etc. Luckily the Internet gods blessed me with good fortune and I was able to receive some pretty awesome advice from a number of sources. Here I will detail all of the advice given, and some take aways I had.

Illustration by KASI MINAMI

Design pattern suggestion( Reddit ): Turn Scarlett’s Listener into a Dbus service and have her core features connect as clients.

I popped my head in over to /r/gnome, and posted this … and /u/totallyblasted really came through for me. Besides helping me w/ my fundamental understanding of what an idle_add is, he presented a concept that blew my mind and really simplified, well, everything. See below:

/u/totallyblasted suggestion( reddit ):

Pattern seems valid on quick glance, except I’m wondering why using idle and not threads. I for one thing question something else. Wouldn’t it be better to create listener as service where clients register over dbus and where your application is the first client for it? Not only would this make reusable design for other applications as well, you’d end up with 2 simpler parts instead of one complex.

Tony Dark reply( reddit ):

Now that is a very very interesting idea /u/totallyblasted[1] re: Listener as service and the application client. Going to play with that idea on paper for a bit … Re: Threads vs idle add, I think I must just have a fundamental misunderstanding of how to call something in a separate thread that has it’s own mainloop ( ie. something using Gstreamer ). In the past, my application only survived one run before there was a stack dump, always thought it was because I was calling things improperly. I’ve definitely read https://wiki.gnome.org/Projects/PyGObject/Threading[2] several times, but perhaps I need to give it another glance over.

/u/totallyblasted suggestion (con’t):

Well, for one thing you’re not handling anything time pressing, since you’re processing voice and just send activation signals. Might be even more interesting if your service is designed to support design somewhat like this (over dbus) SomeApplication > Service - I need to know when user says “shit”, “blabla” or “somethingelse” or chains of words Service > Application - Gives session dbus back and then just signals back which registered string was said on that one whenever those occur Application > Service (later in running) - Add “bleh” to my session keywords and remove “blabla” And so on. Note, that service could serve for any app. And your application would become two things at once. Application as intended and POC for any other application interested in your service Note, that since you’re already in 2 separate applications, most of the otherwise needed threading is already gone. And the fact that you don’t process anything time pressing, but you have to process constantly without breaks makes threading more suitable.

Tony Dark thoughts:

You can follow more of the conversation over on Reddit, but what you’re really looking for is right here: https://github.com/bossjones/scarlett-dbus-poc. I created a Proof-of-concept for this new iteration of Scarlett. Yes yes, I know I could have just created another branch in the main repo, but I wanted to start fresh, get things working, then move everything back into the main repo later after I had a more complete understanding of everything. By opening multiple ssh windows and running scarlett_listener.py and scarlett_tasker.py, Scarlett comes up and starts handling requests immediately. So much cleaner, easier to make modifications to her now! This assumes you have a complete Scarlett environment put together ( Need better scripts * cough Ansible * to handle this … more later ). Here is a quick diagram on how things work using this system:

Diagram of New Scarlett Listener Dbus Service ( Showing only one signal in diagram for simplicity )

Diagram of Listener Dbus Service and Tasker connecting as client  

I also wanted to include some secondary suggestions for improving Scarlett as a result of this blog post from other developers across Reddit+ Twitter.

Suggestion: Upgrade Python 2.7 + Gstreamer 0.10

/u/Eingaica had several questions regarding the versions of Gstreamer + Python I chose to use:

I currently don’t have the time to look at the details of your system, so just a quick question: Why Python 2.7/GStreamer 0.10 and not the current versions Python 3.x/GStreamer 1.x? Also a small note: g_idle_add doesn’t use a separate thread. It just adds a function to the main loop, which will call it (from the main thread) when it’s idle.

Tony Darks Thoughts:

Moving to Python 3 isn’t something I’m completely interested in doing quite yet. Python 2.7 is going to be around for a while and has a lot of support still, so notttt yet… Moving up to more recent versions of Gstreamer, GTK, Pocketsphinx etc though … That is a must. Prerequisite work is a better way to provision a Ubuntu/Raspbian server from start to finish. Looking at Ansible to do that.

Suggestion( twitter ): Implement Scarlett w/ an Actor Model design pattern

https://twitter.com/stommepoes usually has some really awesome suggestions for Scarlett … from features ideas to application design, etc. This time around the suggestion revolved around looking into refactoring Scarlett to use a Actor Model implementation https://en.wikipedia.org/wiki/Actor_model. Currently there has not been any work on testing this implementation out since the POC worked right off of the bat … But I do plan on giving it a look at some point in the near future ( Yay Christmas break ! ) See a snippet of the convo below:

More here:

https://twitter.com/bossjones/status/656195018378629120

https://twitter.com/stommepoes/status/656025216452313088

Tony Dark’s Next steps:

Now that the newly designed Scarlett POC is working, it’s time to finally port over to PyGI, Gtk3, and GStreamer 1.0. Will be using this awesome guide to do that ( https://wiki.ubuntu.com/Novacut/GStreamer1.0 ), but there are a couple prerequisites:

1. Need to have a better way of provisioning a vagrant box w/ all prerequisites for Scarlett from scratch. This allow others to play w/ Scarlett and contribute Pull Requests etc to the various repositories. Ansible seems like a great choice since it only requires SSH access to the machines at your disposal. This will also make configuring things like Raspberry Pi’s much easier. FYI I would like to try moving the test instance to 14.04 or even 14.10 Ubuntu Desktop ( Since Pocketsphinx master claims it supports that off of the bat ). The interesting challenge will be figuring out which parameters we need to change in the newest version of Pocketsphinx (5prealpha) to obtain the same level of accuracy that we had in version 0.8. Reason why this will be a bit of a challenge is because Pocketsphinx 5prealpha was completely rewritten in C++ and v0.8 was written in C. The values we need to alter might be in a different location/renamed/not used anymore…Won’t be the most difficult thing in the world, we always figure it out don’t we?
   
2. Research & Development on running Scarlett in a Docker container. Even easier than provisioning your own Vagrant boxes for development … using docker to simply pass around a prebuilt image will make things like running tests on TravisCI SUPER FAST. The one difficulty will be figuring out how to get a “GUI” application to work within a container … Luckily Jessie Frazelle over at Docker have an awesome blog post on running GUI apps in Docker!
   
3. (Optionally) Get everything working on Ubuntu 12.04 LTS first before switching everything else. I already have a working 12.04 Desktop instance preloaded with everything we need. But by default it is using older versions of … everything. We can still update the code to use PyGI, Gtk3, and GStreamer 1.0 but we might not see the benefits of doing that till we upgrade all other underlying libraries as well ( 12.04 LTS isn’t supported anymore )
   

Decisions decisions! Ill take a second to map everything out and see what to tackle first over the next couple days. If you have thoughts on what makes the most sense, drop a comment below!

As always, thank you for following the journey. I will continue to win … fail … and learn from each step, then relay it to you! If you have ideas for blog posts please let me know!

#3Dprinter update, Z coupler is on + Wiring HELL

‪#‎3Dprinter‬ update. My new Z coupler part finally came in the mail and I got it on correctly ( didn’t use to much super glue his time!!! ). Final task… Which is a difficult one… Find a way to get alllllll of the wires through that cluttered area at the base and connected to the printer chip board. Tried taping it together and threading it through using some zip ties to pull it…. but that didn’t work. Might have to try one at a time next. Tonight might finally be the night mannn.

Decided to build a 3D printer over the weekend! #printrbot

Tony Dark made something! Yay!

So I decided to build a 3D printer this weekend… It’s basically finished but I accidentally super glued my Z coupler + screws incorrectly. Waiting on #printrbot to mail me a replacement, then will finish up and start printing things! Not 100% sure what I am going to print yet, and how it can benefit things like Scarlett, but I’m excited about having the ability to do so. Leave some ideas below of what I should print!

3d Printer: Printrbot Simple 1405.

Calling all PyGObject/PyGI/GNOME Experts: Help Critique Scarlett’s Application Redesign!

IMPORTANT NOTE: This is a letter I’m sending to the python-hackers-list, which is comprised pygtk, pygi, and pygobject maintainers( https://mail.gnome.org/archives/python-hackers-list/ ) I wanted to cross post it just in case someone else has some suggestions for me. Let me know what you guys think in the comments below!

Hi guys,

So I have a somewhat in depth question about the design of my application. Will do my best to just highlight the important information to ensure that there is enough to describe what I’m doing and what my possible pain points might be.

Overview:

I’m working on a home automation daemon call Scarlett. Scarlett is a voice controlled python application that listens all of the time for her name, then listens for a qualifier ( Command / task to do, eg. "What's the time", "turn off lights" ) and performs an action. Once the task is done, Scarlett’s state is reset and the process begins again. Scarlett has several main features:

– Listener: Scarlett’s Listener is a subclass of GObject, and utilizes Gsteamer + Pocketsphinx to handle Speech-to-text. Pocketsphinx has been compiled against python to create a Gstreamer plugin.

– Speaker: Scarlett’s Speaker is also a subclass of GObject, and utilizes Gstreamer + Espeak via gst-plugins-espeak to handle all Text-To-Speech.

– Player: Scarlett’s sound player is also a subclass of GObject and plays static .wav files when Scarlett completes a task, or acknowledges a command, etc. Player uses Gstreamer to play sound within the application.

– Feature Objects: Scarlett Feature Objects include TimeTask, WeatherTask, TvTask, etc. Each of them is a subclass of GObject and has a number of function that allow it to obtain information or complete a given action. More on that later.

– Scarlett Daemon: Scarlett Daemon ( aka the SD ) is the glue between all of her features. It contains the GObject MainThread Loop and acts as a supervisor for all processes run in separate threads. The SD is a subclass of dbus-python module ( dbus.service.Object ), and uses gobject.GObject.connect() to connect signals with handlers within the Listener, Speaker, and Player. An example of this is:

sd.connect('rdy_for_keyword',listener_rdy)

NOTE: Where listener_rdy is defined as from scarlett.listener.listener_rdy_handler as listener_rdy

Hopefully the basics make sense, and if not, I hope they already outline any flaws.

Version information:

Since someone will probably ask this sooner than later:

Python 2.7.10

PyGst 0.10

Gobject (2, 32, 4)

What am I looking for:

I simply want advice on if I am designing my application correctly. I’ve gone through several iterations of it at this point, and this time I really want to make sure everything is architected out property before I write any more code.

Important Notes For People Not Familiar With GObject:

• idle_add is a GObject function that lets you run something in a separate thread.
  
• emit sends a signal to another object and does something based on that.
  
• In the diagrams below, anywhere that says 'IA:' means idle_add.
  
• In the diagrams below, anywhere that says 'E:' means emit.
  

Diagram:

The following 2 pictures show my proposed design for my application.

Without Numbers:

With Numbers:

It is meant to describe from start to finish what it is like issue a command, start a task, then finish. Blue lines represent a GObject.idle_add, and Green lines represent a GObject.emit. Anything that has 2 direction arrows means a signal is emitted when a task is finished to Scarlett Daemon, which is where we run our mainloop / supervisor.

Walking through the process:

=================

1. Scarlett is started, Gobject’s Mainloop is in Scarlett daemon. If you’re curious, here’s the code that handles that here: https://gist.github.com/bossjones/7ce13ade25941b34b63f
2. Using a gobject.idle_add, Scarlett Daemon starts the Listener in a separate thread.
3. Using a gobject.idle_add, Scarlett Daemon starts the Player in a separate thread.
4. Using a gobject.idle_add, Scarlett Daemon starts the Speaker in a separate thread.
5. After all threads have been started, Scarlet Daemon sends a signal to the Listener via a gobject.GObject.emit saying 'listener_rdy' to let it know that it should start running Gstreamer which will bring up a long-running Speech-To-Text process.
6. Scarlett Daemon does a gobject.GObject.emit() saying 'play_sound_ready' to the Player thread to announce that everything is up and running. This starts a short lived Gstreamer run that simply plays a beep sound to acknowledge that the task happened. The Player thread then emits a signal back to the Scarlett Daemon telling it 'play_sound_fin'.
7. Scarlett Daemon emits signal to Listener thread 'rdy_for_keyword', this alerts the Listener to begin listening specifically for keyword: 'Scarlett'.
8. After 4 failed attempts Listener emits 'max_listener_fails_reached' to Scarlett Daemon to restart the entire process. Failed attempts are stored in a GObject property named 'stt_failures' ( default value: 0 ). If it find a correct match, it emits 'keyword_found' to the Scarlett Daemon instead and sets GObject property 'stt_failures' back to its default 0. Listener sets the GObject property 'kw_found' to 1 to keep the state of the object.
9. Scarlett Daemon emits 'play_keyword_found' to the Player thread which kicks off a gstreamer run that simply plays a beep sound to acknowledge that the task happened. When finished playing, Player emits 'keyword_found_fin' to Scarlett Daemon.
10. Scarlett Daemon, which already emitted 'keyword_found' to the Listener, now emits 'rdy_for_cmd' to the Listener thread to let it know we are ready to find a qualifying command. eg 'what time is it', 'turn on the lights', 'turn on the tv', etc.
11. After 4 failed attempts Listener emits 'max_listener_fails_reached' to Scarlett Daemon to restart the entire process. Failed attempts are stored in GObject property 'stt_failures'. If it find a correct match, it emits 'cmd_found' instead and sets GObject property 'stt_failures' back to its default ‘0’.
12. Scarlett Daemon receives 'cmd_found' signal and brings up a feature object in a new thread using an gobject.idle_add. These objects, like everything else in this project is a subclass of GObject ( ex. TimeTask, WeatherTask, TvTask, etc ).
13. The correct command task is run, and emits 'task_action_fin' to Scarlett Daemon when finished. ( eg 'tv_poweron_fin' )
14. Scarlett Daemon emits 'scarlett_speak' to Speaker thread which kicks off a Gstreamer run using gst-plugins-espeak to perform a Text-To-Speech.
15. When Speaker is finished running, it emits 'speaker_fin' to Scarlett Daemon.
16. Scarlett Daemon emits 'play_sound_complete' to Player thread which uses Gstreamer to play a wav file as acknowledgement of job being finished. It emits 'play_sound_fin' when done to Scarlett Daemon
17. Scarlett Daemon emits 'scarlett_run_fin' to Listener which tells it to reset all GObject properties back to their default.
18. Listener emits 'listener_reset_fin' to Scarlett Daemon when finished.
19. Loop and repeat!

Sorry if any of that was dragged on too long, but wanted to make sure I wrote that out as clear as possible.

My questions:

=================

1. Does this sound like a valid design pattern for my application and will it work with the python GObject bindings? I understand I might need to add a couple threading objects to prevent things from running at the same time ( threading events, locks, rlocks, etc )
2. Does the long running Listener thread which does Speech-To-Text in the background make sense and will it work as intended?

Thanks for taking the time to read this friends, hope to hear from you soon!

– Tony Dark

Scarlett: Micro Blogging + Keeping An Inventors Log Book

Hey friends! Back with a quick blog update for ya. For several years now i’ve been a big fan of Evernote and their Web Clipper ability. Super useful especially combined with a premium account and an iPad mini ( offline mode FTW! ). But I soon ran into an issue when I noticed this during my python/gstreamer research:

ZOMG, 314 ITEMS?!?!!?

I use that web clipper like a mad man!!! Sometimes I just clip things just to clip it! That made it super difficult to figure out where I left off in my R&D for Scarlett and what articles I should be reading next to help further my journey.

While re-watching this scene from Iron Man, it struck me, I needed a journal to keep status updates of important discoveries throughout this process:

Yeah yeah yeah, I know Tony Stark is merely talking to Jarvis and Jarvis is acting as his personal journal, but Scarlett doesn’t quite have that feature yet :)

Till then, I’ve decided to create a separate twitter account dedicated strictly to Scarlett development. Introducing @scarlettapp

Tweets by @scarlettapp

This will help as I continue to iterate on Scarlett and discover new strengths/weaknesses in her code base!This will especially become useful in the upcoming major change involving moving from Gstreamer 0.10 -> 1.0 and GTK+2 -> PyGI/GTK+3.

Make sure you tune in there if you want to know what’s new! NOTE: Don’t worry I’ll still be tweeting from @bossjones as well.

Look out for another blog post this week on setting up a python virtualenv with PyGI/GTK+3/Gstreamer 1.0 !

Till next time…..

– Tony Dark ( and #gastonthefrenchie, new assistant )

PS Check out these cool videos too:

• XRobots - Iron Man Cosplay J.A.R.V.I.S. Electronics System Overview, for my life sized suit, costume

Scarlett’s Brain: Managing State + Events Using Threads + Redis + Gevent

So I was shooting the shit with my buddy @jackyalcine the other day, and he brought my attention to a couple of other very cool home automation projects that are out there, eg api.ai and wit.ai. I also decided to open up hubot on github just for the hell of it to see how they were dealing with sharing resources, states, and events throughout objects. I realized several things:

1. I noticed that a lot of these projects are implementing their own DSLs. Whats a DSL? According to this [stackoverflow post]:

Domain-specific language (DSL) is a programming language or specification language dedicated to a particular problem domain, a particular problem representation technique, and/or a particular solution technique.

2. I also noticed that these same applications are using a PubSub design when implementing their applications to alert different objects, services, etc of changes to the state of the application, eg “Are the lights on or off right now”, “When was the last time pocketsphinx failed to recognize a keyword”, etc etc.

I started to realize that Scarlett is lacking in those departments. Currently it is not easy to define a new service that Scarlett can integrate with. Now… going the route of a DSL might be a little overkill for what we’re trying to do right now, there are some steps can be made to define what a very basic service in Scarlett should be able to do( Especially since things are must more object oriented now, yay inheritance ). So i’m left to figure out one more thing … How do you pass around information regarding the state of Scarlett and everything she interacts with? Whenever she’s rebooted, she has no idea if the lights are on/off, if she should be processing a task in the background or anything! What should we do?

Enter the Publish-subscribe Pattern.

What does it mean?

According to wikipedia

In software architecture, publish–subscribe is a messaging pattern where senders of messages, called > publishers, do not program the messages to be sent directly to specific receivers, called subscribers. > Instead, published messages are characterized into classes, without knowledge of what, if any, subscribers > there may be. Similarly, subscribers express interest in one or more classes, and only receive messages that > are of interest, without knowledge of what, if any, publishers there are.

Pub/sub is a sibling of the message queue paradigm, and is typically one part of a larger message-oriented > middleware system. Most messaging systems support both the pub/sub and message queue models in their API, e.g.> Java Message Service (JMS).

This pattern provides greater network scalability and a more dynamic network topology, with a resulting > decreased flexibility to modify the Publisher and its structure of the data published.

I will now begin working on creating an Event Emitter for Scarlett. Each object will be able to subscribe to a Redis backed channel and will be able to publish their current state, subscribe and see if there was a change in their state, if a keyword was identified by [pocketspinx], any type of failures that occurs, etc. All of Scarlett’s services( brain,listener,hue lights,window blinds,sonos,forecast) will now be fully aware of what the other is doing. This will also be helpful as we move towards the greater goal of splitting up different tasks onto different nodes. Since we’re using Redis, Scarlett can now pick up right were she left off after a system reboot, yay for persistence! Lucky for us, the amazing redis-py module has an awesome PubSub class which will allow us to do this all very easily.

While thinking about this new design approach, I automatically thought about a potential bottleneck we might encounter…. Listening on the redis channel is a blocking operation, meaning, by default only one object can access it at a time, and that’ll prevent the rest of the application from running. How do we solve that?

gevent to the rescue !

What is gvent?

gevent is a coroutine-based Python networking library that uses greenlet to provide a high-level synchronous API on top of the libev event loop.

What else can it do?

• Fast event loop based on libev (epoll on Linux, kqueue on FreeBSD).
• Lightweight execution units based on greenlet.
• API that re-uses concepts from the Python standard library (for example there are Events and Queues).
• Cooperative sockets with SSL support »
• DNS queries performed through threadpool or c-ares.
• Monkey patching utility to get 3rd party modules to become cooperative »

Basically, by using gevent, we’ll be able to achieve long polling for Scarlett, and allow her to periodically query for changes in the state of anything she’s connected to. AWESOME!

balloob over at github has an awesome home automation project where he identified a very smart and straight forward architecture diagram for his project, check out this image below:

We will be implementing something very similar for Scarlett. Pretty soon we’ll be able to move into multiprocessing in Python, boy oh boyyyyy! Get ready for the next wave of awesomeness!

I’m going to be using the following articles as resources to implementing the ScarlettEventEmitter:

• Gevent, Long-Polling & You
• Concurrent connections to Redis with gevent and redis-py

As always, you can keep up with what’s going on by checking out the Scarlett repo on github!

– Tony Dark

Guess who’s back? ( again ) - Scarlett TIME

Friends! I’m back !!!!

As some of you know, the past couple months have been pretty crazy between stuff at the office & working on getting a new apartment etc. Well guess what, life has finally chilled out a bit and I can rededicate some time to my favorittttte girl, Scarlett.

What’s different this time around?

1. I bought an apartment in NYC! Why the hell does this matter? Well since I’m a homeowner now, I can rip up any floor I want, change any fixtures I want, etc etc. We now have the creative freedom to do whatever we want with Scarlett. Nice!

2. I’ve learned A LOT about Python over the past couple months from working on other projects at behance/on-the-side. This means debugging Scarlett code should be much faster now ( sheesh how did I do anything without virtualenv, ipython and auto-complete ), and i’m adding something great … TESTS! Yes, all Scarlett code will now be tested in travis.ci. Ill start off with unit tests, then later ill implement some integration tests, and a couple more. Going to take a little time to get everything green, but this will ensure that I don’t keep breaking stuff by switching code around all over the place. I’ve completely refactored the original repo to make it look a lot more like a python application. 

3. I’m segmenting things out more & using the right languages to get every job done. The feature list for Scarlett has gone through the ROOF. So many things I want to add all over the place. This new version of Scarlett will introduce a Plugin/Features class which will allow you to chose which features you want to active via your config file ( forecastio, spotify, hue lights, etc etc ). This means you wont have to use features if you don’t want to, simple configuration for the win! Also, you’ll be able to extend and create your own Plugins! I’m also going to be embracing C a lot more for certain tasks…speed speed speed. Couple ideas for things like zeromq, flask etc that I have… but I’ll save that for a future post. 

4. Loose coupling to save the day. Each “part” of Scarlett will be made into a separate service. I like the idea of having several scripts doing one task VERY WELL, instead of having one gigantic script. Easier to debug + optimize if we take this approach as well. Pick this up at work, and its been helping us out with handling some very BIG projects across the board.

5. Documentation. Im writing down every damn thing. Good practice and easier for others to contribute.

6. Exploring multiple HW devices. I’m still working with the Raspberry Pi, but also experimenting with the Beaglebone, an old Thinkcentre, and my Virtualbox dev machine. I’m doing this to compare performance metrics based on system specifications etc. Even playing around with the concept of creating a Beowulf cluster.

More updates coming, get ready for awesomeness!

– Tony Dark

Improving Scarlett’s Performance on Raspberry Pi w/ C + Gearman Part 1

So believe it or not…Scarlett is having some performance issues/bottlenecks. Insert funny innuendo joke here. Everything works AMAZING on my Ubuntu 12.04 LTS Virtualbox image … What could be the reason behind this? Let’s talk about some theories.

“Raspberry Pi has limited resources anyways.”

How much can one really do with 512MB SDRAM and a 700 MHz ARM1176JZF-S core CPU anyways right? Not exactly… Clearly the Raspberry Pi is an extremely popular device with a TON of support from the community . There are some really really cool projects out there already! I’ve had a long conversation with one of my buddies on Facebook ( E.COLES  ) regarding this, and we went back and forth on whether or not Scarlett just needed a better body to handle some of her new features. While that might happen in the near future, I think the Raspberry Pi is the perfect device for the time being. I have my eye on a couple other mini computers including the Cotton Candy for when it becomes time to integrate Opencv .

“The bottleneck is just in my Python code.”

This is certainly possible for a number of reasons. First of all Scarlett is doing A LOT of work. She’s using both Gstreamer and Pocketsphinx bindings to listen and translate A LOT of data into text. On top of that, she needs to use a large number of python modules to parse this data and do something with it, eg connect to WolframAlpha or Forecast.io’s API. Most importantly, I am super SUPER new to Python and its “best practices”. Everyone who has ever picked up a couple of programming languages know that it takes a little time to figure out the “Right” way to do things, and the “wrong” way. I’ve used a couple of Python benchmarking tools to help me identify potential slow performing functions etc, and cleaned up a large number of them … Regardless, A lot of this will be alleviated as Scarlett gets closer to becoming a public project, but again, there are a lot things that will need to happen before that… Major refactoring of her code, simple deployment system to get all of her dependencies installed/configured for people who want to support the project ( stay tuned for that one ), unit tests, etc.

So how do we get some better performance on the Raspberry Pi ?

Break.Down.The.Problem

Proposal:

How about instead of running both pocketsphinx and gstreamer all through Scarlett’s Python listener class … split out the pocketsphinx portion, modify some c, run pocketsphinx_continuous in the background and send all identified speech to text ( stt ) jobs to a Gearman server which will then be picked up by a python listener worker who will parse all data to see if it’s a correct command or not??

“Ummmm…………. wtf did he just say?”

Let make this simpler with a little diagram.

This is how Scarlett current works. Everything is managed by Scarlett’s Python Listener class.

Click to zoom

This is my proposed way of doing it.

Click to zoom

Why the hell would you want to do that?

First of all the idea came to me via Nathan Broadbent and his awesome voice activated Microwave Pi Project. All of this was done using pocketsphinx , Sinatra , A Raspberry Pi and an Arduino Nano .

From his videos, it seems as though his microwave speech recognition performs very well, and listens all the time for new voice commands / executes them in a timely manner. How is it he was able to get RUBY to perform all of that work faster than a Python Application? There are studies that show that even though Ruby is an AWESOME language, its average runtime can be a bit slower than Python ( source ). While diving into his work I came across this: pocketsphinx_microwave.c . No way… Nathan modified some pocketsphinx code to interface directly with his ruby application, see the following lines:

He then runs it in the backround with this mw_voice_control which is a simple bash script that runs:

This calls pocketsphinx and runs all speech recognition.

Awesome.

For those of you who have never played with C, a quick explination of why we would want to touch such an old ass language:

C is an imperative (procedural) language. It was designed to be compiled using a relatively straightforward compiler, to provide low-level access to memory, to provide language constructs that map efficiently to machine instructions, and to require minimal run-time support. C was therefore useful for many applications that had formerly been coded in assembly language, such as in system programming.

That means its FAST AS HELL.

But, what’s up with Gearman though?

Couple reasons behind using Gearman, i’ll take an excerpt from Benjamin Wooton’s answer on quora :

A. You wish to increase the reliability of your system by persisting your queues of work to disk within some broker, rather than keep them hanging around in memory where they are vulnerable to a system crash;

B. You need to introduce asynchronicity in your system, returning control to the user whilst you queue and carry out other tasks in the background.

Process jobs in the background, work on them as you find them in the queue, do it all very quickly.

So what does this mean now? What are you going to do ?

Couple things in the works right now:

• Going to modify continous.c file to include a call to gearman which will send all recognized words in the background to a Gearman task called scarlett_commands. Pocketsphinx will act as a Gearman client.
• Going to comment out all calls to gstreamer and pocketsphinx from within the listener class and instead change it into a Gearman worker who simply connects to the Gearman server and waits to process jobs we define.
• See if scarlett begins performing better

This will be part of a undetermined part series. Spoiler alert I already have the C client and python worker running together, but wanted to do a tiny bit more work before writing up a new post.

Hope you guys are still with me on this journey, we’re literally going to play with EVERY type of technology by the end of this!

May the Wave be with you,

– Tony Dark

Note: As always, I will come back and fix my stupid grammar later.

Scarlett Update: Control TV via pyserial + Arduino Uno

NOTE: THIS BLOG POST IS A COUPLE WEEKS OLD BUT I DIDNT TAKE THE TIME TO READ THROUGH ALL OF IT AND POST IT IN A TIMELY MANNER. ANOTHER BLOG POST COMING SOON!

What’s up family, been a while, but wanted to let you guys know I’m still actively working on everything. Couple important updates I wanted to make you aware of.

My biggest goal at the moment is getting my Arduino Uno to control my Apple TV (x2), Toshiba 42" LED TV, and my AC Unit. Once I can do this regularly from some testing platform, I’ll integrate it into Scarlett’s voice recognition system on my Raspberry Pi.

SPOILER ALERT … I’ve succeeded!! But figured out a couple things I need to fix/tweak/recognize to make this system a little more robust. Lets start at the beginning of this feature.

NOTE: For the testing of this feature, I've been using my [ Macbook Pro ] to develop on, simply because I can move around from room to room and test, etc.

How did I approach this?

Day in and day out at Behance, I work in a terminal window trying to connect clusters of servers together, in an effort to make them talk to each other, run tasks, etc etc. Most of the time, I accomplish this by using some sort of Command Line Interface tool … so it seemed like the appropriate approach to take w/ this TV feature. I wrote a quick CLI tool called ScarlettTV CLI in Python . ScarlettTV CLI connects to your Arduino via a serial port ( I used the pyserial library btw ) and runs arbritary code on your Arduino.

Research

There’s a million different blog posts out there on teaching your Arduino how to decode/send IR signals to something ( Special thanks to my brother Joshua Arenas for the crash course on how to use an [ Arduino Uno ]! ).

My favorite ones are by Ken Shirriff , who created the awesome Arduino-IRremote library :

• Blog post - Controlling your stereo over the web with the Arduino infrared library

And Cyborg5 , who has refactored Ken Shirrif’s code and extended it with IRLib :

• Blog post 3 out of 3 - IRLib Tutorial part 3a: Sending IR Codes

Read any of their blog posts to figure out how I was able to decode all of the remotes in my house. They’re very well written and extremely descriptive. I will post another blog post later on with details on how I setup my Arduino to send IR signals complete with an up to schematics, diagrams, etc once i finalize its design, currently I don’t like how many wires I have lying around, want to compact things a bit and make it sleeker & a bit more portable.

Btw, if you’re thinking about doing write ups about your Arduino project, make sure you check out Fritzing ( [ ITS SUPER WAVYYYY ], B ).

According to their website:

Fritzing is an open-source hardware initiative that makes electronics accessible as a creative material for anyone. We offer a software tool, a community website and services in the spirit of Processing and Arduino, fostering a creative ecosystem that allows users to document their prototypes, share them with others, teach electronics in a classroom, and layout and manufacture professional pcbs.

Example diagram:

• http://fritzing.org/projects/arduino-presenter-receiver

Playing with ScarlettTV CLI

As I mentioned, I’ve been doing a lot of this project in Python … please bare with me cause I’m still learning how to do things the “Python way”. Eventually I’ll sit down and go through the correct guides when I refactor everything, but for right now my focus is results, not pretty code. :) Feedback is certainly WELCOME though, tell me how to make things CLEANER.

NOTE: From what I understand, you should ALWAYS develop in Python using Virtualenv to keep your projects and their dependencies separate. A perfect guide on doing this can been seen here via the guys at [ Digital Ocean ]: https://www.digitalocean.com/community/articles/common-python-tools-using-virtualenv-installing-with-pip-and-managing-packages

Using pip I installed the following on my Macbook Pro ( NOTE: Not my virtualbox setup yet ) before starting anything:

pip install pyserial==2.7
pip install argparse==1.2.1

Going to need both of these in order to make the python script run correctly.

Running it with the –help flag will tell you the basics.

malcolm@malcolm-MacBookPro1 [feature-parse-launch*] ~/dev/scarlettPi/scripts: ./scarletttv_cli --help
usage: CLI tool to control Arduino IR signals. [-h] [--doctest] [-v] [-V]
                                               [-d {frizzytv,appletv,toshiba}]
                                               [-c COMMAND] [-t] [-p PORT]
 
optional arguments:
  -h, --help            show this help message and exit
  --doctest             perform doc tests
  -v, --verbose         be verbose in all things, go with god
  -V, --version         show version
  -d {frizzytv,appletv,toshiba}, --device {frizzytv,appletv,toshiba}
                        set device to one of several ( eg. frizzyTV )
  -c COMMAND, --command COMMAND
                        Set command
  -t, --test            Run serial port test suite
  -p PORT, --port PORT  Set the serial port
malcolm@malcolm-MacBookPro1 [feature-parse-launch*] ~/dev/scarlettPi/scripts:
    

If you run it with the –test flag, the cli tool will print a list of available ports.

malcolm@malcolm-MacBookPro1 [feature-parse-launch*] ~/dev/scarlettPi/scripts: ./scarletttv_cli -t
/dev/cu.Bluetooth-Modem
/dev/cu.Bluetooth-PDA-Sync
/dev/cu.Bluetooth-Serial-1
/dev/cu.Bluetooth-Serial-2
4 ports found
malcolm@malcolm-MacBookPro1 [feature-parse-launch*] ~/dev/scarlettPi/scripts:

After identifying the IR codes for all of my remotes, I made use of Python’s powerful dictonary data structure ( In other languages, this is simply an associate array ). I stored all of my device names, their commands, and the proper IR Hex code associated with sending that command. Lets take a look at the Apple TV section in depth a bit more.

command_master_list = {
      "appletv": {
        "up": "1,77E1D030,32",
        "down": "1,77E1B030,32",
        "left": "1,77E11030,32",
        "right": "1,77E1E030,32",
        "menu": "1,77E14030,32",
        "pause": "1,77E17A30,32",
        "play": "1,77E17A30,32",
        "circle button": "1,77E1BA30,32" 
      },

So again, device name we’re interested in is appletv.

If you look at an Apple TV remote ( 3rd Generation ), they’re designed to be very simply like most of their products. They only have a couple of different button options: up,down,left,right,menu,pause/play,circle button/enter. That’s it! I decoded all of their hex codes and place it into a string. To read the command format, the first number indicates the protocol number of the IR signal, which is NEC in this example. The second number is the hex code to be transmitted, and the final number is the number of bits ( 32 bits for all Apple TV commands ).

To see more examples of this, please check out Cyborg5’s IRserial_remote.py example on github.

IMPORTANT ASSUMPTION: Right now we’re assuming your Arduino has already been pre programmed to send IR signals. I modified Cybog5’s IRserial_remote.ino sketch to work for my needs. Again, we will go over this in more depth in another blog post focused more on the Arduino.

Now that we have those defined, I simply run with the following flags while my Arduino is connected to my MBP and pointed directly at my Apple TV:

./scarletttv_cli --device appletv --command left

And boom, I’m now controlling my Apple TV from the command line!

Next post…turning this functionality into a python class that scarlett can call directly.

I’m BACK! Ready to work on Scarlett! New Features!

Hey guys,

Super sorry I’ve been M.I.A guys. Two of my buddies ( #Dwalk + #Savage ) mentioned to me the other day that I made so much of a big deal about everything involving this project… then just dropped it all….Things got really busy and I had to refocus on a couple of things, but I'm back and in full force! Over the past two days, i’ve been working on Scarlett again, and i’ve made some huge progress.

NOTE: When you working on something big, but you come to a wall…step back…look at old code with new eyes, and a new approach, and you’ll come up with something.

The Black Tony Stark is back:

—

New Features

API integrations? Voice controlled Hue lights? You asked for it, I did it. Here’s a couple videos showing it in action:

Pocketsphinx + Philips and Hue Lights Integration:

—

Lets try changing the colors of the lights now …..

—

Gave her a voice using Festival ( Chose this over eSpeak ):

—

Integration with forecast.io, now I know the weather every morning:

Do you guys have any ideas for me?

Any feature integration ideas?

Please let me know, I want more idea!

Upcoming feature … TV integration ( Control the TV w/ my voice ).