Music

The past week or so have been busy - small tasks here and there, but the main things have been music related.  

Scarborough Faire (http://www.srfestival.com/) is a Renaissance festival located just outside of Waxahachie, TX.  I've bee associated it for the past 20 years in various capacities, but primarily as the performing company music director and lead musician for quite a while.  Now this isn't hard, but it is time consuming at times, especially in March right before we open.  First off we have workshops on the weekends, which I go down at least one day to do music related stuff.  I usually direct the performing company in full ensemble music (the end of day "Parting Glass") and also work with the instrumentalists and pub sing director to make sure things are lined up.  there's also some other non-music stuff, but mustly music.

Next on the scarborough list is some outside rehearsals.  normally we can get by with the rehearsals during the workshop periods, but due to weather and a shortened workshop season we've had to have a few more to get some of the newer folks up to speed.  it also lets me hear how things will sound, correct some of the music, and get to know everyone better.  While some of these people i've been working with for a long time, others are relatively new. So far we've had one rehearsal and will have at least one more this week, with possibly another next week if called for.  

Now this music doesn't just magically appear for the musicians.  I've got to go out and find music we can use.  This involves sourcing the music, transcribing it, getting it into a book form, and storing it in a format that everyone can get to.  I've been a big fan of Finale (http://www.finalemusic.com/) for transcription, but because I'm on a slightly older versionI can't really create a good book from here directly.  I print out the individual sheets for each tune, and then copy and paste an image into Microsoft Word to do the final book format.  a quick print (or now, save to PDF) and I can copy it up for others to make use of.  I've been working on this book off and on for about 10 years, mostly as a hobby, and it has been used at numerous other faires and venues from here to Kansas City.  A consistent look is nice, since it does make our music look more professional.  previous to this most faire related music was photocopies of photocopies, with different sizes, wordings, and degrees of readability.  

The other part is the Pub Sing book, which is different than the tune book, though the process is the same.  I take all of the suggestions that we come up with for new songs, and start vetting them.  again, we cannot use modern songs, or stuff that is still under copyright, since we don't really have a budget for that.  there is plenty of stuff out there, though - it is just a matter of finding it.  Mudcat.org is one of the better sources of folk music lyrics, and Google and the internet have made research a lot simpler.  Finding a song that was published at least back in the 1800s is a good sign, and further back is even better.  I also try to make sure our music is reasonably unique from other music groups out there.  We don't want to be singing the same thing as anyone else, even though everyone has their own spin on how they will do a song.  

From here I take a look at the lyrics.  some tunes we can make use of directly, with no changes.  others, because they are from a later time or location will have mention of things that are currently unknown in Scaroborough in the 1530s, such as Captain Kidd from the 1750s, or New York City.  if we can change things to something more period we do so, otherwise the song gets chucked to the side.  I also have to look at the tune, and try to avoid some of the more common recordings that people are familiar with, since that could be infringing on things.  usually if I can source it back to some of the original works in the area, such as O'neals book of irish songs, or the Child ballads (the writer is named Child - a lot of the music is not suitable for children) or the ROUD bibliography, then I'm pretty sure we have a good song.

the next steps, which we'll do later this week, is get all the lyrics together into a book and see how bit it is.  We print, fold, and bind things ourselves, so our books need to be a multiple of 4 pages (a single page folded front and back) and a single song can't go over a page turn, though it can be on two facing pages.  a bit or rearranging, some creative space management, and we'll have a book ready to print.

The other stuff I've been doing is playing a couple of gigs at, of all places, a haunted house.  Dark Hour (http://darkhourhauntedhouse.com/) is a local haunted house with a world class director.  It's been open for about a year, and is still going strong.  Now one of the things you do when you have a house like this is make sure you can do more than be open for Halloween, so they have multiple themed events throughout the year, and Saint Patricks day is one of them.  and what is Saint Patricks day without traditional music?  so Friday and Saturday night a small group of us got together and kept people entertained as they waited to enter the house itself.  Irish step dancers, music, and a fun venue.  And afterwards on Saturday I actually had a chance to go through the house.  And if you get a chance to, go.  There is a diffference between the venues that put together something for the short term versus the permanent houses.  the displays are better, the rooms are thought out, the staff well trained, and there is a continuity and a story line along with the theme.  

Our musical group consists of three people, and we've played together multiple times.  I'm actually the newcomer to this group - one of their members moved out to New York a while back - so i'm learning some of their arrangements, sopplying some of my own, and seeing how my mix of talents will fit in.  we had a couple of practice sessions over the past few weeks, just enough to take the raw edges off a few things, but we did have a couple of decent shows.  

Now it is Tuesday, March17th, and believe it or not, I'm not doing anything musically related right now this evening.  It's reasonably quiet, I had a good dinner, and I'm enjoying it.  I've been in ensembles before that are performing tonight.  once I had three gigs on one day, another I lost a fiddler right before the show due to a death in the family, and spent the next hour calling every musician I knew to get some fill-in.  we survived the first, and scrounged enough stuff to make up for the fiddle tunes.  But tonight it is peaceful.  

Tomorrow it is back to song preparation and lyric checkout.

A Change in Direction

One nice thing about being independent is the capability to do something different when you decide to.  Since I've reached a bit of an impasse with the spam filter, I'll play with another task I've been banging around with for a while.

But first some background information, and a short introduction to amateur meteorology. 

For the past 20 years or so I've been tracking the weather a little bit closer than most people.  In the back yard on a heavy metal stand is a weather station, which allows me to see the conditions in my yard at pretty much any time.  This is a Davis system (http://www.davisnet.com/), which consists of the outdoor component, which measures rainfall, temperature, wind, solar conditions, and so forth, and an indoor base station that records and displays the measurements.  This isn't my first station, though.  I have worked my way thorugh a system from Oregon Scientific (http://www.oregonscientific.com/us/) which did essentially the same thing.  I switched about 9 years ago when I was changing servers around, and the new one had no serial ports.  Oregon Scientific at the time only supported serial communications, while the Davis system supported USB.

Now this information os good, but gets really useful if you can pull it up on your web browser, or in an app on your phone.  For that I have been transmitting this data to the Weather Underground (http://www.wunderground.com/) for pretty much all of that time.  You can see my particular weather station at http://www.wunderground.com/personal-weather-station/dashboard?ID=KTXMCKIN31

Now there are plenty of ways to get this information from a base station uploaded.  There are software packages that you can run on a laptop, from Ambient and Davis.  These are nice, customizable, and sort of supported.  I had been doing that until last summer, when the laptop I was using as a server started having problems, and being unable to move the software easily to another laptop, I started looking at other options.

In the past few years the concept of a network appliance has cropped up.  this is usually a little device running Linux or some other lightweight operating system, usually with a few ports, powered by a 5 volt source, WiFi, powered by a very common 5 volt power source, and geared towards a single application.  As I was searching for replacement software I stumbled across a number of weather appliances that were capable of supporting most of the consumer weather stations out there.  After researching them a bit, I went with the Meteobridge unit (http://www.meteobridge.com/wiki/index.php/Main_Page).  I was able to get it running in a couple of hours, setting up the base station, Wunderground push, WiFi, and other items, and decommission the old laptop.

The Meteobridge acts as a bridge between your weather station and most of the weather services out there.  It is capable of communicating over USB (or even ethernet) to your weather station, and then pushing this data to all of the services you wish to push it to.  It also has its own web interface that allow you to configure and monitor it.

Now one of the things it is capable of doing is to FTP the data to another location.  It makes use of a template that you supply and stores the data at a location that you can specify.

So quite a while back I created a template and got it set up to store the data on an FTP folder on a laptop in the house.  I haven't really had time to do anything else with this since then - at least until now.

So what I'm planning on doing is to make use of this data.  broadcast the information out on Facebook, Twitter, or whatever, update web sites, send out alerts, and generally be a communications hub for exporting this data.  Now you might ask, "Doesn't the Meteobridge applicance already do this?", and the answer is "sort of".  It will fire off twitter messages, and email things based on some conditions or periodically, but that is about it.  it is real good about talking to weather services, but not so well on the social scene.  Think of the meteobridge device as a weather nerd.

So what do I want to do with this?

Fire off emails when conditions warrant it

tweet about weather in the area

post to facebook periodically

update a web site with the information

So I started creating a base application this afternoon.  I already had my template, and it was working fine.  An hour spent in Visual Studio got me enough scaffolding to read the incoming data (dumped conveniently in an XML file) and toss it into a database.

So now what?

First I'm going to figure out the rules part.  given the current information, I would like to determine if a message should be sent.  a rules engine of some sort will be needed. 

Now there are three ways to handle rules.  The first is to hard code the rules, and compile it into the application.  I'm against this because every time you want to do something different you'll have to rebuild the entire application.  It also precludes being able to distribute the application and make it useful to others in any meaningful way, since what they might consider as important to send an email about won't even show up on your radar.

Second is some sort of database to hold the rules, with numerous tables and fields containing the data for the various rules.  While this is more flexible than the previous, it doesn't allow someone to deviate too far off the reservation.  No complex rules, no deviation from the patterns already set.

the final way is to use a rules engine.  Essentially what a rules engine is is a library that can parse a set of rules, and then run the rules on the incoming data, firing off if one of the rules triggers.  Now this cold be done by hooking into the current compiler, but that is overkill.  instead I'll make use of the rules engine within the Windows Workflow Foundation.  This will allow me to look at the data and act upon it as needed.

Communication is next.  I'll need ways to communicate with the various social applications out there in some meaningful way.  Most of them have interfaces that are published, so that should not be too hard.  Email is also a solved problem, and will just be a matter ot working with a library to get things out there.

I'll need a user interface of some sort to manage the rules, and possibly the rest of the system parameters.  I don't know the format of this right now, but it could be a Webforms project, WPF, web based, or some combination or permutation of the above.  

Finally, I'd like a web front end to display the data on, though this isn't really needed.  I can always get to the latest data on Wunderground, but sometimes you want to see slightly rawer data.

So thats it for now.  As I creat this thing I'll drop my observations in here, and once it get it into a more usable condition I'll put it up on one of the open source repositories.

Poll or Watch

I've been periodically going back to the email portion of the categorizer I've been playing with.  on one hand I can poll the server for information, and on the other I can wait and have the server tell me that things have changed.  

I created a version that would poll the server for changes, but I'm not really liking it.  Polling takes effort on the client side to get things done, and most of the time there will not be any changes.  While it is handy to get everything, it just doesn't feel efficient.

Now under the IMAP specs, you can only watch one folder at a time, which means that changes in other folders won't be caught.  While it would be nice to see changes in multiple folders, you can't do it under a single connection.  

And thats the loophole I think I can exploiit.  I only want to watch a couple of folders - the inbox and the spam folder.  If i could open multiple connections to the mail server, I could do this.  and according to the IMAP specs I am allowed to do this.  one of the primary reasons and differences for IMAP over POP was it would allow multiple clients to access the server at the same time, allowing unique IDs, and generally allowing a constantly open connection to the server.  

So that's what I'm going to do.  I'm going to rebuild things a bit to support multiple connections to the server,  rather than a single connection and trying to manipulate everything through that.  The two connections will point to inbox and spam respectively, and will be able to watch for the changes in these folders.  

Inbox will watch for incoming new emails.  when one is received, it will be scored, and if it passes the threshold, will be moved to the spam folder.  If an email is deleted, then train the application for a good email.  If the email is moved outside of the spam/inbox folders, then it is considered to be good, and trained as such.

the spam folder will also be watched.  when an email is deleted, train it as a spam message.  if moved to the inbox, ignore it.  if moved someplace else, then train as a good email.

I'm also pulling back the scope a bit.  originally I wanted to allow multiple categories, and move things to different folders based on this categorization.  Normally this wouldn't be a problem if I could maintain all of the different categories with the email, and have the user acknowledge all of the categories.  but email can only go in a single bin at the same time - if you open your email client you'll notie that you can't have the same email in separate folders.  you can copy the email, but that is a copy - not the original email.  Since I don't wish to disrupt the normal email reading workflow just to categorize it in multiple places, I'll stick with the binary on Spam or No Spam.

Another problem is training an email that has more than one valid category.  if the user hasn't checked all of the categories that the email is in, then it could end up be miscategorized, hampering training.  Also the problem of tracking what categories that an email falls into.  do we just choose the best one, or dump all of the possible categories into the headers in some form or fashion.  

So if we just go with the best, we can train on this category, ignoring all of the others.  basically it would be trained on the best category, without training on the other categories.  This would work, but will slow the training quite a bit, and require more emails to train.  Finding the best category will also be problematic, since we will probably need to get a percentage on each one, and then select the best one.  

another way to possibly get to a single category is a series of binary categorizations, much like a search tree.  at the top of the tree we would categorize possibly between spam and non-spam.  next down in the non-spam side might be a series of different categories, possibly getting to finer and finer detail.  

For now i'll just do the simple yes or no on whether it is spam.  If that works, then I'll expand it out to more explicit categorization of email.  It still won't solve the multiple category problem, but will at least be a consistent way of looking at the email.