Post Sabbatical

I've been back at it for two weeks.  Two weeks of getting settled into a new position, learning new faces, and getting exposed to a different culture and environment.  I'm going for a large company to a much smaller one, from many different clients and locations to a single product suite and a single location.  It'll take some getting used to, but the differences aren't as large as you think.

In a larg company, how many people do you interact with on a daily basis?  Ten? Twenty?  And how many clients at a time?  These numbers ar probably a lot smaller than you might think. I usually only had a single client I had to make happy at a time, and most teams were small.  No where near the thousands of people in the company.  So I'm actually working with a larger proportion of the company. My new management team is in the same building, and I do see them daily.  It is a different world.

I'm getting used to the commute.  For years my longest drive was usually the one to the airport.  Once at a location I was at most ten or fifteen minutes from the site.  Right now it is a comfortable thirty minutes, which allows me to easily drop my daughter off at school, get to work, get tngs done, and make it home in time to pick everyone up, make dinner, and relax in the evening.

So what have I learned in the past eight months?  Well, the first is I'm not a prolific blog writer.  I should do more of that, and may do so in the future as time permits. Topics will change and very s bit more, though.

I have learned a few technical things.  A bit of self study, a seminar on the Microsoft campus, some personal projects to dig into some different concepts.  I was able to get more depth on a couple of topics, and some breadth over the Microsoft development environment.  There is so much out there that it is almost impossible to know everything, but sometimes just knowing what is there is good enough, if you can get the details later when you need to use it.

House work takes time.  Whether it be cleaning, cooking, washing, or whatever.  My wife usually did that, but was busy with her final semester and board exams to do that.  I still missed things, though.  Cooking was not a problem, but the cleaning never lets up.  My wife has grabbed back most of that, but I still do the cooking - I actually enjoy it, and according to my wife, I'm better at it than her.

I got to spend plenty of time with my family.  Biking to school with my daughter, planning events and other family activities, even a vacation to central Florida for an extended time.  Being on the road sort of isolate me a bit fm that, though FaceTime, Facebook, and other communications eased that a lot.  

Finances were someone I had to watch a bit.  Since we had no direct income during that time, I cashed out some stock to cover expenses. Some luxuries were dropped (except for that vacation) and cutting corners here and there helped.  Cooking at home, cheaper entertainment, and just not doing some things were some easy things to do.  

So where do I go from here?  Well, first will be to integrate myself into my new work environment.  So far that is relatively painless, probably due to my experience with going to a new client periodically.  Getting used to going to a work place every day hasn't been too hard - I was up every morning at the same time anyway to get everyone to their respective schools, and tried to keep busy most of the day, though I was able to sneak in a nap here and there.

Wel, that's it for today.  I'm helping a friend with a production of Macbeth, and need to get to my place for some music pieces coming up in the fourth act.

Locking down the toaster: IoT security

You just got your new toaster set to deliver toast at exectly the right shade of brownness.  Going to bed with the knowledge in mind, you wake the next morning, shuffle into the kitchen, where the pile of burnt and raw toast meets you.  Did the toaster fail? is it possessed?, or did your new internet enabled toast have a visitor from an external source?  For now, lets go with the latter and assume that someone is wanting to play with your toaster, someone that you dont want to place with your toaster.

How do you prevent this, you might ask.  well, you could unplug the toaster, move to a cave, and promise to never get a newfangled internet toaster again.  the second, and probably less drastic, is to protect your toaster from external influences.  this is called (wait for it) security.

Security is the act of protection, whether it be aggression (i'm going to attack now, to prevent you from atatcking me later) to passive (padlocks on the doors, bars on the windows) to monitoring (i know someone broke in, and here's a picture).  we're going to focus more on the later items, those of authentication, authorization, and logging.

Authentication is the act of figuring out who someone is.  in the real world you can do this in many ways, such as look out the peephole to see whoe is at your door, ID badges in the work place, or keys for locks.  the first of these, looking out the peephole, is a form of positive identification.  you have to know who they are before you open the door.  the second is also a form of this, but with the addition of a token, you can identify someone that you don't know.  think of the phone guy coming to your door.  he's got a badge, but he's also not your aunt martha.  the final is a key for a lock.  here the user of the key is anonymous, in that you don't have to know who the holder of the key is, but it is sufficient to know that they are allowed because they are holding the key.

in the computer world ther eare plenty of ways to authenticate people.  passwords, number tokens, and fingerprint sensors help to do this.  there is some element of trust to setting up the authentication in the first place, but once its done, you are golden.  Now this can be done in many different places, and in many different forms.  A lot of applications handle the authentication on their own, keeping track of users and passwords internally, while others will make use of various third party authentication services from Google, FaceBook, Twitter, and plenty of others.  

Authorization is what you are allowed to do after we know who you are.  Our hypothetical toaster might let anyone in the family toast bread, but only one person change the level of brownness.  One way to look at this is a list of users, a list of capabilities, and a bunch of lines connecting the people to what they can do.  This is a simplified way of looking at things, but thats ok at this time.  

Logging allows you to see what is going on, by tracking who is doing what activity, as well as other properties on the system.  Logging would tell us that someone is cooking toast, and possibly when it is being done, and who is doing it.

When we start applying security to our internet of things we start seeing that there are a couple of places that we may need to apply it.  first is at the device, since we don't want unauthorized people cooking toast.  second is on a hub or server that the toaster may be communicating with.  in this case think of the device as the thing that needs to be authenticated and authorized.  

I'll stop here for today.  there are plenty of ways of do these tasks, and i'll beat them to death later on.

The Internet of Things, or, The Toaster says Hello

The Internet of Things (IoT, for short) is a concept that has been around for over 15 years, and with the advent of cheap devices and plenty of bandwidth, has started to gain some traction.  So what is it?  There are a lot of definitions out there, covering a lot of area (and even more marketing hype) but there needs to be a good way of stating what it is.

 

So here is my definition:

 

The Internet of Things is a way of looking at how mundane devices can communicate with each other.

 

Yes, it's short, probably a lot shorter than you would expect.  But at the heart of it, it is things communicating with other things.  So maybe a few more definitions are in order

 

A Device can be anything that generates, processes, or consumes data. 

Communication is the act of sending and receiving of information

 

So far it sounds a lot like the regular internet.  You've got devices communicating back and forth, and you've got information flowing around between devices (in this case computers and servers).  We need that final word - Mundane.

 

Mundane - average, ordinary, not standing out in any way.

 

So we're not talking about connecting a tablet to eBay.  We're talking about connecting your front door to the security system, your mailbox sending a text message, or your toaster to the breadbox.  Some of these can already be done, and if you follow the news, you are seeing more products out every day that communicate back and forth.

 

But there is something lacking, and that is the ability of any device being able to communicate with any other device (within reason, of course - some conversations just don't make sense).  I've got the capability the tie my mailbox into the security system, and from there send me a text message whenever it is opened, but I can't just take that mailbox sensor and tag it directly from a browser.  I've got to go through an intermediary or two before I can get to the information.  Now this in itself isn't a problem, since there are times when you want this sort of aggregation to take place.  But in my case, since this information is local to my house, why should it go from the mailbox, to a centralized panel, to a server someplace in California, to a text message aggregator, and then finally I get that buzz on my phone when the postman does a delivery? 

 

Each device also has its idiosyncrasies for setting them up and communicating with them.  A WiFi camera will use 802.11x to communicate, while a 2Gig system uses a different set of frequencies and protocols to get things done.  Even the same class of devices on the same network may not communicate in the same way, meaning you cannot just get a device and add it to your security system, but have to get a specific type of sensor, sometimes with proprietary protocols that prevent it from working with other devices. 

 

So let me add another definition:

 

Open - the systems use non-proprietary methods of communications using commonly available protocols, and the communications medium is transparent as to function and capabilities.

 

So, a camera will look like a camera, and behave the same, at least at this level.  This isn't to say that a camera manufacturer can't add more functionality that can only be gotten to in their own special way - just that if you are a device, you have to play nicely.

 

So what am I doing here?  Well, I can't leave well enough alone. I don't do the hardware side, so developing devices isn't really my thing.  I'm an architect and developer, and I see a lack in the overall area.  There has been some mentions of a Web of Things, but it is also a bit nebulous.  So I'm not planning on creating the next widget to make my toaster talk to the breadbox.  Instead I'm exploring how I can make it easier for things to talk to each other.

 

In later posts I'll continue this discussion with myself.  There's a lot of stuff out there that I can look at, and maybe someone or something has already solved my problem.  But I'm guessing not, since my toaster still doesn’t recognize me and toast things the way I want them to.

A Break from my Break

If you have been following this at all, you may notice that there is a bit of a gap between the last post and this one.  Three months, in fact.  Quite a bit of time when measured against the internet, but miniscule when compared to the age of the universe, so everything is relative.   

What have I been doing?  oh, quite a few things.  Taking a few classes in Windows Azure, mowing teh yard, getting people through college and licensing, playing music, and doing some recording, just to name a few.  I also took a nice long vacation and did some heavy self inspection.

Since the first I've also started getting back into the work swing of things.  Formatting my resume for the first time in ages, registering on various technical job boards, setting my criteria for what I wished to do, and contacting places, while politely telling the recruiters with their perfect opportunity in New Jersey that I wasn't willing to relocate, and that Houston is too much of a commute from where I live now.  Five calls yesterday, ten the day before, and an undetermined number before then.  I've been limiting my search to within a ten mile circle (which given my location on the edge of the DFW Metroplex, cuts out a lot of area) to cut down my commute, and primarily looking at senior developer and architect jobs.  Drawing that line cuts a lot of the positions from Dallas, Irving and Addison, but even so there are quite a few things up in the Plano, Frisco and Richardson area.  

We're still living off of savings, primarily from selling stocks purchased over the past fifteen years.  It is sort of like retirement, but different.  I have mentioned that watching expenses is an important thing that you will need to do on a sabbatical, and that is still ongoing.  We have had some backsliding, such as eating out a bit more, but I still enjoy cooking at home.  

We did take a vacation through most of June, though.  Orlando, Disneyworld, Universal, NASA, and the beach for an extended period of time.  while portions did cost a bit (You can't eat at Disney too cheaply) we did save quite a bit by using hotel points, airline miles, and car rental points from my previous travels to cover the really big stuff.  From there it was primarily food, admission, and souvenirs.  

I'll see what the next few weeks brings.  I've talking to a few people about open positions that they have, and am entertaining a few others here and there.  There isn't a dearth of positions, and I got time to be a bit choosy, so I'll choose whats best for me.  

Finances

One thing about taking a sabbatical is handling the inflow and outflow of money.  Now in most cases there will be no inflow - i.e. you aren't working, therefore no inflow.  So how do you balance the inflows and outflows so that they balance, at least in the long run?

the first thing is your budget.  Budgets aren't just for when you are working.  you'll also need one for managing your expenses for the upcoming period.  You can think about this as a precursor for retirement.  

My budget is simple.  I've got a house, a car, and a family.  I've got a mortgage, I need to maintain things, and we need to eat.  Also, we need health insurance, auto insurance, and a few fun things here and there.  So my budget needs to handle that.  

Mortgage and insurance are the biggest ticket items.  Mortgage is around $2800, and insurance around $1400 a month. auto insurance is around $50 (paid every 6 months), gas and car maintenance runs a few hundred, and amusements will run a bit here and there.

Now I could save on the insurance, but I like not having to worry about that sort of things.  I could go through the marketplace and get insurance for around 800, but with a high deductable.  I extended my coverage under COBRA, which let me continue my existing insurance at the same rate.  It is interesting to see the cost of health insurance through your employment - most places that supply insurance usually have pretty decent programs, since a sick employee isn't adding any value.  

The mortgage will not change.  I've got a pretty good rate, so I don't really want to refinance.  Most of the insurance offers are currently above that rate, so refinancing will actually have me paying more over the long run.  

auto insurance, gas, maintenance, and the like will not change, either.  I can drive a bit less (and have done so) get cheaper insurance, and skimp on maintenance.  But here in Texas there isn't any such thing as walkable shopping, so a car is essential.  Gas is pretty much a constant - a mile driven will consume x amount of gas.  Biking to a few places helps to cut that a bit. Getting a more effficient car would lower that, but at the expense of a car payment.  Since both of our cars are paid off that would be an extra $300 that we can delay on.  Maintenance could also be delayed, but usually the longer it is put off, usually the more it will be.  Insurance could also be downgraded, but stuff happens.  Even a minor fender bender will cost thousands, and having made a claim for this just last month, I'll keep that right where it is.

Food is an easy one.  Restaurants are easy, seductive, fast, not too nutricious, and don't seem to cost too much.  but added up over time the cost of food when you eat out all the time adds up.  By preparing more meals at home I can cut that expense down quite a bit, and make them more nutritious.  I shop around, get good ingredients, and stock up a few things when they go on sale.  Most of the meals I prepare are simple - pasta, vegetables, some sort of protein, and dessert every once in a while.  I usually cook a bit extra, since I can reheat stuff for another meal with little added cost.  While we don't eat every meal at home, even a few a week cuts the expenses by a good percentage.  I eat most lunches at home, rather than eating fast food.  

Entertainment is a tougher one - We don't spend massive amounts of time going to amusement parks or at the movies, but we don't cut them out completely.  We still go to the movies, usually a smaller Cinemark down the road from us that has a cheap matinee.  We don't go to Six Flags, but are planning a trip to Florida in June.  

Again, it comes down to planning.  I went through this exercise before I decided to take my break, I have been saving money in a couple of locations, such as the stock purchage plan and some external investments, and factored in what I would need on a monthly basis.  I planned periodically to draw off money from selling stock.  Keeping these draws spread out keeps the expenses down, and makes me think more of where the money is going.  

Thinking about this in terms of retirement, there will be only a few differences.  The mortgage will be gone, and the health insurance will be offset a bit by medicare/medicaid.  there will be some income in terms of pensions, but the draws from a 401K and other sources will still remain about the same.  I'll figure out some of that later on, exactly.  Beffoore that time I'll think about whether we'll want to move because of taxes.  I'll always have federal taxes, but different states have different state income taxes, property taxes, and the like.  Texas has no state taxes, but has one of the higher property taxes.  Louisiana has one of the lowest, but has a higher income tax.  essentiall every state has advantages and disadvantages, not all of them monetary.  access to shopping, entertainment, health care, and other amenities are also important.  Personally I like some space around me, but I also like close by shopping and high speed network access.  In twenty years these priorities may change, though.

Two Months

So far I've been on my sabbatical about two months.  I've watched the last snows of winter (such as they may be down here in Texas) move into the budding trees of spring.  I've spent plenty of time around the house doing some of the various chores that I've been putting off for a while, and even some of the longer range ones.  I've even started some grass seed in a couple of the dead spots, since I know that I will be here for a while to water it until it is started.  I've even mowed the yard a couple of times, when it started looking a bit shaggy.

I spent a day moving a hot tub.  Now this isn't quite as arduous as you might think.  What I have is a large SofTub, which is about 6 feet across and can easily be moved/dragged/tugged around.  two people make it move faster, though.  Now the location we had it at was right off of our patio, which was right off of the bedroom.  due to some poor planning on my part, i neglected to take into account the noise factor of a hot tub kicking on in the middle of the night.  after a few nights of this I did put a switch on there to prevent it from coming on after 10, but even then, I knew I would have to muve that sucker.  I already had a place picked out - just on the other side of the patio, about fifteen feet away.  

Moving the hot tub involves three major steps. First I have to move the hot tub to a completely unrelated location.  this is so I can do the second step, which is move the cement blocks to the new location.  Finally I could move the hot tub to the new location.  After a couple of minutes hooking up the pump (the SofTub has an external pump, which also acts as a heater) and the rest of the day filling it with a garden hose I was ready to turn it on.  Even then I couldn't get in for a while because the water was a balmy sixty degrees, which for me is a bit cold.  it would take a couple of days to warm it up fully.

Getting grass to grow is my next task for that area.  I have a decent lawn, I take care of it, but this is a hundred square foot area of dirt.  A trip to Lowes later I had grass seed, and shortly after that had it spread out and watered.  I'll have to do that for a few weeks until it takes hold good, and then the regular sprinklers can do their job.

I've been teaching my daughter how to rid a bike.  She was a complete chicken for the longest time, but after it finally annoyed me enough I removed a training wheel.  She adapted and complained.  After a couple of weeks I removed the other training wheel.  She complained again.  finally this past weekend I got her on the bike and we kept working at it until she could traverse the length of the driveway - about sixty feet.  And after a few more days of practice, a scraped knee or two, and a couple of bumps, we rode our bikes to school, and will ride them back this evening.  She's about to outgrow her bike, though.  I think the Easter bunny might have to bring a new one since her birthday is still three months off.

So where will the next few months lead? I've got a few things planned to do, so here's the list.

1) Scarborough Faire (http://www.srfestival.com/) - I'm the performing company music director, and as such, need to be out there a good chunk of the time.  I've been doing this quite a while, and it is worthy of another blog entry a bit later.

2) Learning - I'm planning on taking a couple of classes in Azure and possibly a few other things.  some of these land on the weekends, but that is normal.  I'll work them in as I can.

3) Get my wife through her final semester - She's been working on the Dental Hygiene program at Collin College for the past two years.  She's taken her board exams a couple of weeks ago, and has six weeks of classes left.  just doing the chores around the house helps a lot there.

4)  Personal Projects.  the weather watcher, the mail categorizer, and a couple of others creeping about in the back of my head.  I've started some of this stuff, and have also taken the time to rebuild and update my local server machine.

5)  Get back to work - I like what I did, but I needed the break.  As I discovered when I started this I didn't need a change in venue as much as I needed to take care of family stuff for a while.  I might try something new, or go back to the old - that hasn't been written yet.

6) Update my resume -  I usually update my resume out on the various job sites about every three or so years, and it's due time.  I liken it to feeding time in the shark cage when I toss my resume out there.

7) Vacation - I owe my wife a vacation.  we've been talking about it for a while, and once everyone is out of school for the summer (graduated, entering second grade, or whatever) we want to go someplace.  We're thinking Disney World, since everyone likes that, but we'll spend some time there this go around, possibly seeeing some of the other sites such as NASA and Busch Gardens.

Music

The past week or so have been busy - small tasks here and there, but the main things have been music related.  

Scarborough Faire (http://www.srfestival.com/) is a Renaissance festival located just outside of Waxahachie, TX.  I've bee associated it for the past 20 years in various capacities, but primarily as the performing company music director and lead musician for quite a while.  Now this isn't hard, but it is time consuming at times, especially in March right before we open.  First off we have workshops on the weekends, which I go down at least one day to do music related stuff.  I usually direct the performing company in full ensemble music (the end of day "Parting Glass") and also work with the instrumentalists and pub sing director to make sure things are lined up.  there's also some other non-music stuff, but mustly music.

Next on the scarborough list is some outside rehearsals.  normally we can get by with the rehearsals during the workshop periods, but due to weather and a shortened workshop season we've had to have a few more to get some of the newer folks up to speed.  it also lets me hear how things will sound, correct some of the music, and get to know everyone better.  While some of these people i've been working with for a long time, others are relatively new. So far we've had one rehearsal and will have at least one more this week, with possibly another next week if called for.  

Now this music doesn't just magically appear for the musicians.  I've got to go out and find music we can use.  This involves sourcing the music, transcribing it, getting it into a book form, and storing it in a format that everyone can get to.  I've been a big fan of Finale (http://www.finalemusic.com/) for transcription, but because I'm on a slightly older versionI can't really create a good book from here directly.  I print out the individual sheets for each tune, and then copy and paste an image into Microsoft Word to do the final book format.  a quick print (or now, save to PDF) and I can copy it up for others to make use of.  I've been working on this book off and on for about 10 years, mostly as a hobby, and it has been used at numerous other faires and venues from here to Kansas City.  A consistent look is nice, since it does make our music look more professional.  previous to this most faire related music was photocopies of photocopies, with different sizes, wordings, and degrees of readability.  

The other part is the Pub Sing book, which is different than the tune book, though the process is the same.  I take all of the suggestions that we come up with for new songs, and start vetting them.  again, we cannot use modern songs, or stuff that is still under copyright, since we don't really have a budget for that.  there is plenty of stuff out there, though - it is just a matter of finding it.  Mudcat.org is one of the better sources of folk music lyrics, and Google and the internet have made research a lot simpler.  Finding a song that was published at least back in the 1800s is a good sign, and further back is even better.  I also try to make sure our music is reasonably unique from other music groups out there.  We don't want to be singing the same thing as anyone else, even though everyone has their own spin on how they will do a song.  

From here I take a look at the lyrics.  some tunes we can make use of directly, with no changes.  others, because they are from a later time or location will have mention of things that are currently unknown in Scaroborough in the 1530s, such as Captain Kidd from the 1750s, or New York City.  if we can change things to something more period we do so, otherwise the song gets chucked to the side.  I also have to look at the tune, and try to avoid some of the more common recordings that people are familiar with, since that could be infringing on things.  usually if I can source it back to some of the original works in the area, such as O'neals book of irish songs, or the Child ballads (the writer is named Child - a lot of the music is not suitable for children) or the ROUD bibliography, then I'm pretty sure we have a good song.

the next steps, which we'll do later this week, is get all the lyrics together into a book and see how bit it is.  We print, fold, and bind things ourselves, so our books need to be a multiple of 4 pages (a single page folded front and back) and a single song can't go over a page turn, though it can be on two facing pages.  a bit or rearranging, some creative space management, and we'll have a book ready to print.

The other stuff I've been doing is playing a couple of gigs at, of all places, a haunted house.  Dark Hour (http://darkhourhauntedhouse.com/) is a local haunted house with a world class director.  It's been open for about a year, and is still going strong.  Now one of the things you do when you have a house like this is make sure you can do more than be open for Halloween, so they have multiple themed events throughout the year, and Saint Patricks day is one of them.  and what is Saint Patricks day without traditional music?  so Friday and Saturday night a small group of us got together and kept people entertained as they waited to enter the house itself.  Irish step dancers, music, and a fun venue.  And afterwards on Saturday I actually had a chance to go through the house.  And if you get a chance to, go.  There is a diffference between the venues that put together something for the short term versus the permanent houses.  the displays are better, the rooms are thought out, the staff well trained, and there is a continuity and a story line along with the theme.  

Our musical group consists of three people, and we've played together multiple times.  I'm actually the newcomer to this group - one of their members moved out to New York a while back - so i'm learning some of their arrangements, sopplying some of my own, and seeing how my mix of talents will fit in.  we had a couple of practice sessions over the past few weeks, just enough to take the raw edges off a few things, but we did have a couple of decent shows.  

Now it is Tuesday, March17th, and believe it or not, I'm not doing anything musically related right now this evening.  It's reasonably quiet, I had a good dinner, and I'm enjoying it.  I've been in ensembles before that are performing tonight.  once I had three gigs on one day, another I lost a fiddler right before the show due to a death in the family, and spent the next hour calling every musician I knew to get some fill-in.  we survived the first, and scrounged enough stuff to make up for the fiddle tunes.  But tonight it is peaceful.  

Tomorrow it is back to song preparation and lyric checkout.

A Change in Direction

One nice thing about being independent is the capability to do something different when you decide to.  Since I've reached a bit of an impasse with the spam filter, I'll play with another task I've been banging around with for a while.

But first some background information, and a short introduction to amateur meteorology. 

For the past 20 years or so I've been tracking the weather a little bit closer than most people.  In the back yard on a heavy metal stand is a weather station, which allows me to see the conditions in my yard at pretty much any time.  This is a Davis system (http://www.davisnet.com/), which consists of the outdoor component, which measures rainfall, temperature, wind, solar conditions, and so forth, and an indoor base station that records and displays the measurements.  This isn't my first station, though.  I have worked my way thorugh a system from Oregon Scientific (http://www.oregonscientific.com/us/) which did essentially the same thing.  I switched about 9 years ago when I was changing servers around, and the new one had no serial ports.  Oregon Scientific at the time only supported serial communications, while the Davis system supported USB.

Now this information os good, but gets really useful if you can pull it up on your web browser, or in an app on your phone.  For that I have been transmitting this data to the Weather Underground (http://www.wunderground.com/) for pretty much all of that time.  You can see my particular weather station at http://www.wunderground.com/personal-weather-station/dashboard?ID=KTXMCKIN31

Now there are plenty of ways to get this information from a base station uploaded.  There are software packages that you can run on a laptop, from Ambient and Davis.  These are nice, customizable, and sort of supported.  I had been doing that until last summer, when the laptop I was using as a server started having problems, and being unable to move the software easily to another laptop, I started looking at other options.

In the past few years the concept of a network appliance has cropped up.  this is usually a little device running Linux or some other lightweight operating system, usually with a few ports, powered by a 5 volt source, WiFi, powered by a very common 5 volt power source, and geared towards a single application.  As I was searching for replacement software I stumbled across a number of weather appliances that were capable of supporting most of the consumer weather stations out there.  After researching them a bit, I went with the Meteobridge unit (http://www.meteobridge.com/wiki/index.php/Main_Page).  I was able to get it running in a couple of hours, setting up the base station, Wunderground push, WiFi, and other items, and decommission the old laptop.

The Meteobridge acts as a bridge between your weather station and most of the weather services out there.  It is capable of communicating over USB (or even ethernet) to your weather station, and then pushing this data to all of the services you wish to push it to.  It also has its own web interface that allow you to configure and monitor it.

Now one of the things it is capable of doing is to FTP the data to another location.  It makes use of a template that you supply and stores the data at a location that you can specify.

So quite a while back I created a template and got it set up to store the data on an FTP folder on a laptop in the house.  I haven't really had time to do anything else with this since then - at least until now.

So what I'm planning on doing is to make use of this data.  broadcast the information out on Facebook, Twitter, or whatever, update web sites, send out alerts, and generally be a communications hub for exporting this data.  Now you might ask, "Doesn't the Meteobridge applicance already do this?", and the answer is "sort of".  It will fire off twitter messages, and email things based on some conditions or periodically, but that is about it.  it is real good about talking to weather services, but not so well on the social scene.  Think of the meteobridge device as a weather nerd.

So what do I want to do with this?

Fire off emails when conditions warrant it

tweet about weather in the area

post to facebook periodically

update a web site with the information

So I started creating a base application this afternoon.  I already had my template, and it was working fine.  An hour spent in Visual Studio got me enough scaffolding to read the incoming data (dumped conveniently in an XML file) and toss it into a database.

So now what?

First I'm going to figure out the rules part.  given the current information, I would like to determine if a message should be sent.  a rules engine of some sort will be needed. 

Now there are three ways to handle rules.  The first is to hard code the rules, and compile it into the application.  I'm against this because every time you want to do something different you'll have to rebuild the entire application.  It also precludes being able to distribute the application and make it useful to others in any meaningful way, since what they might consider as important to send an email about won't even show up on your radar.

Second is some sort of database to hold the rules, with numerous tables and fields containing the data for the various rules.  While this is more flexible than the previous, it doesn't allow someone to deviate too far off the reservation.  No complex rules, no deviation from the patterns already set.

the final way is to use a rules engine.  Essentially what a rules engine is is a library that can parse a set of rules, and then run the rules on the incoming data, firing off if one of the rules triggers.  Now this cold be done by hooking into the current compiler, but that is overkill.  instead I'll make use of the rules engine within the Windows Workflow Foundation.  This will allow me to look at the data and act upon it as needed.

Communication is next.  I'll need ways to communicate with the various social applications out there in some meaningful way.  Most of them have interfaces that are published, so that should not be too hard.  Email is also a solved problem, and will just be a matter ot working with a library to get things out there.

I'll need a user interface of some sort to manage the rules, and possibly the rest of the system parameters.  I don't know the format of this right now, but it could be a Webforms project, WPF, web based, or some combination or permutation of the above.  

Finally, I'd like a web front end to display the data on, though this isn't really needed.  I can always get to the latest data on Wunderground, but sometimes you want to see slightly rawer data.

So thats it for now.  As I creat this thing I'll drop my observations in here, and once it get it into a more usable condition I'll put it up on one of the open source repositories.

Poll or Watch

I've been periodically going back to the email portion of the categorizer I've been playing with.  on one hand I can poll the server for information, and on the other I can wait and have the server tell me that things have changed.  

I created a version that would poll the server for changes, but I'm not really liking it.  Polling takes effort on the client side to get things done, and most of the time there will not be any changes.  While it is handy to get everything, it just doesn't feel efficient.

Now under the IMAP specs, you can only watch one folder at a time, which means that changes in other folders won't be caught.  While it would be nice to see changes in multiple folders, you can't do it under a single connection.  

And thats the loophole I think I can exploiit.  I only want to watch a couple of folders - the inbox and the spam folder.  If i could open multiple connections to the mail server, I could do this.  and according to the IMAP specs I am allowed to do this.  one of the primary reasons and differences for IMAP over POP was it would allow multiple clients to access the server at the same time, allowing unique IDs, and generally allowing a constantly open connection to the server.  

So that's what I'm going to do.  I'm going to rebuild things a bit to support multiple connections to the server,  rather than a single connection and trying to manipulate everything through that.  The two connections will point to inbox and spam respectively, and will be able to watch for the changes in these folders.  

Inbox will watch for incoming new emails.  when one is received, it will be scored, and if it passes the threshold, will be moved to the spam folder.  If an email is deleted, then train the application for a good email.  If the email is moved outside of the spam/inbox folders, then it is considered to be good, and trained as such.

the spam folder will also be watched.  when an email is deleted, train it as a spam message.  if moved to the inbox, ignore it.  if moved someplace else, then train as a good email.

I'm also pulling back the scope a bit.  originally I wanted to allow multiple categories, and move things to different folders based on this categorization.  Normally this wouldn't be a problem if I could maintain all of the different categories with the email, and have the user acknowledge all of the categories.  but email can only go in a single bin at the same time - if you open your email client you'll notie that you can't have the same email in separate folders.  you can copy the email, but that is a copy - not the original email.  Since I don't wish to disrupt the normal email reading workflow just to categorize it in multiple places, I'll stick with the binary on Spam or No Spam.

Another problem is training an email that has more than one valid category.  if the user hasn't checked all of the categories that the email is in, then it could end up be miscategorized, hampering training.  Also the problem of tracking what categories that an email falls into.  do we just choose the best one, or dump all of the possible categories into the headers in some form or fashion.  

So if we just go with the best, we can train on this category, ignoring all of the others.  basically it would be trained on the best category, without training on the other categories.  This would work, but will slow the training quite a bit, and require more emails to train.  Finding the best category will also be problematic, since we will probably need to get a percentage on each one, and then select the best one.  

another way to possibly get to a single category is a series of binary categorizations, much like a search tree.  at the top of the tree we would categorize possibly between spam and non-spam.  next down in the non-spam side might be a series of different categories, possibly getting to finer and finer detail.  

For now i'll just do the simple yes or no on whether it is spam.  If that works, then I'll expand it out to more explicit categorization of email.  It still won't solve the multiple category problem, but will at least be a consistent way of looking at the email.

The (e)Mail is Here

Most people, when they start their email client, are happy when it has no problems and they see new email.  Me, I want to do something with it.  I'm not going to write a better email client, but I do want to make it more useful.  So to make this happen I've been looking at how to better categorize incoming emails, such as spam.

So how do you do this?  there are a number of steps that need to be done before you give an email a thumbs up or down, so if you're not into this sort of semi-technical stuff, you might want to go find a cat video for a while.

so what happens when you start your email client?  To you it looks like it shows you your inbox, and you're happy.  behind the scenes theres more that happens, each with its own points of failure and quirks.  For my example below I'm working with the IMAP protocol.  there are plenty of others out there (POP3, Exchange, Lotus Notes, and so forth) but they work about the same, more or less.

First off, your client connects to the server.  for this it needs to know the server address, the particular port, and what security protocol to use. The address is pretty much like an address for some building - it tells where the server is located.  now there is a lot of other magic out there for taking "imap.google.com" and getting to the correct server that holds your email, but that is for later.  The port specifies what to connect to at that address.  think of it as a particular office in the building that your address is for.  This building has a lot of offices, all of them with some purpose, most of which we don't care about right now.  We want to go to office 993 and only there, since whatever is in that office will know how to talk to us and vice versa. the final bit is the security protocol.  while you could communicate back and forth in plain text, this isn't really recommended.  Basically the protocol allows you to encrypt the text going back and forth between you and the server.

And by text, I mean human readable text (in most cases).  if you could look at the data that is being sent back and forth, you will see that it looks like a set of command line interfaces.  in fact, if you were so inclined you could use a terminal emulator and talk directly to the email server, manage your email, and read and write emails directly.  I prefer an email client, though.  Things are so much simpler that way.

So after we connect, the server would like us to tell it who we are.  The user ID and password are used for this.  now because we are connecting securely, this information is passed in as readable text.  You are connected securely, aren't you?  and once you are authenticated, you're ready.

Well, up to a point.  now you probably need to see what is out there.  usually your client will now get a list of all the directories and subdirectories out there, so that it can present those in a way pleasing to the user.  there are a number of special directories, such as trash, the in box, and so forth that we'll come back to later.

Now that we know what the directories are, we will want to know what is in them.  So your client will usually open the folder you are interested in, read the list of emails, and manipulate them as it needs to.  Sometimes it will maintain a list of emails locally, sometimes it will only work with the ones on the server.  in our case we'll assume that it only gets the list of the header information.

Now you, the user selects an email.  Your email client will opens the folder, retrieves the requested email, marks it as read, and closes the folder, before formatting and showing the email to you.  Deleting an email follows the same pattern - open the folder, do the action, close the folder.  Now you can keep the folder open, as long as you remain in the folder, and your email client probably does that, at least until you change to a new folder.

Now email is pretty much a pull technology - it waits for a request from the client before the server responds with the data.  it is also single threaded in that you cannot have multiple requests for data at the same time.  there are some push models out there (for instance, IMAP supports a Subscribe/Idle set of commands) but it is still pretty much one connection, one line of inquiry.  

So how can we monitor multiple folders at the same time?  IMAP will only let you have a single folder open at a time - opening multiple folders won't hurt anything, but it also probably won't give you what you want. 

the first way is opening multiple instances of the connection to the server.  This would allow each folder to be monitored separately and on its own connection.  this does have the advantage of keeping everything  separate, but then you have to manage multiple connections.  you also have the disadvantage of some servers only allowing a small number of connections open at a time.  keeping all of these connections active will take up resources that would be better used elsewhere.

the second is polling the server periodically to see if there are any changes.  this had the advantage of only keeping a single connection open, but the onus is on the client to poll the server to see what is new.  

A hybrid approach could be used.  We could poll the folders periodically, but use the Subscribe/Idle to have the server push changes in one of the folders (like the inbox) at us as soon as they happen.  

So I started with the subscribe model on my categorizer.  After a couple of false starts, and a reading of the RFC for IMAP, and realizing that what I was wanting was not within the capapbilities of the IMAP protocol (hey, i want to subscribe and monitor all folders, not just one) I backed off of this and went to a polling model.  while this is working I will probably implement a version of the hybrid approach later this week so it will be more respoonsive.  

the watcher will keep an eye on the inbox, mainly to watch for new emails arriving.  as each email arrives it will run it through the categorization algorithms and move it to a new folder.

the polling will watch all of the folders to detect changes, primarily to find the moves and deletes.  a deletion will signify that the email was categorized correctly, while a move will usually mean that it wasn't.  Since i'll only be training when the email is deleted, or possibly moved to the trash, I'll need to detect  these correctly.

So, todays takeaways - 

Mail libraries cover most of the details, but not everything, and especially not the process.

Sometimes you have to go back to the IETF RFC for something to see how it works.

Just because you can send a command doesn't you should send the command.

Everything was simpler back when this stuff was first done, at least in hindsight.

Snow Day

Today was a snow day for most of the North Texas area.  For those of you in the far north, a snow day is something that happens when the area gets so much ice and snow that schools and businesses cannot open.  In our area this is usually about an inch of ice.  It's not that we're a bunch of wusses down here, but people just don't know how to drive on this stuff.  A typical ice storm down here will usually have overturned pickup trucks all over the place, because if you have a four wheel drive pickup jacked up with 18 inch risers so you can drive through mud, then a little bit of ice shouldn't be a problem.  

There's a particular interstate exchange down the road a ways called the High Five.  it is where I-635 and US-75 meet, and is monstrous.  The primary ramp from I635 west to US75 north is uphill, and presents itself nicely to a small hill just to the west of it, a hill which is heavily populated by every news crew in town every time inclement weather comes along, each of them waiting for the inevitable bunch of over ambitious but undertalented drivers trying to go uphill in an ice storm.  usually there will be a couple of people at the bottom, stuck there until things thaw.  But this isn't a deterant to some of those yahoos.  Nope, they say, I'm a better driver than those idiots down there - all I have to do is gun it a bit and i'll coast on up this hill.  I just watch them bump into each other on the way up and down like a mechanical version of a lava lamp.

We also aren't used to preparing for this sort of event.  A typical Sunday at the local grocery store will be almost empty, with a few cashiers and stockers moving about, and the non-church crowd getting things done and out of the way before the mega-churches start letting loose.  This particular Sunday had full parking lots, pull stores, and every single cashier running at full steam.  I was in there to get a roast for dinner, so i could spend some time observing.  There were some normal shoppers in there - they were the ones that had a reasonably full cart, containing what looked to be a normal mishmash of stuff.  then there were the panickers - they were the ones with six loaves of bread, toilet paper, and bottled water, while carrying their lattes.  and then there were the other prepper types, that were stocking up on the beer and munchies, because if you were going to be trapped someplace for a while, you may as well be fat and drunk.  And this was repeated at all of our local grocery stores.  There was mention of shortages on the news that evening, to my amusement, along with the obligatory truck wrecks.

Tomorrow is also a snow day.  It didn't get warm enough to melt the ice, so no major melting.  and anything that did melt is now going to freeze even harder.  while they do treat the main roads with salt and sand, once you start getting into the neighborhoods the only movement is kids sledding down the streets, and idiots in pickup trucks going to make their name trying to drive uphill at the High Five.  

So I stayed home, along with my daughter and spouse, all of which are off from their respecttive schools.  I mainly kept an eye on everyone, watching a marathon of how it's made shows on the science channel.  Tuesday will be more of the same, but possibly with a field trip to a fast food restaurant for a chili dog.

I'll probably dig into GitHub a bit.  i took a look at it yesterday, and I do need to put some of my stuff out on a repository in some form or fashion.  I'll also finish digging into the MailKit library a bit more, just to see how the sequence of events fires off, and then start wiring it together with the bayes filter I was working on Friday.  If all goes well I might have something that works by Wednesday.  Heck, even if it all blows up I might still have something by then, just not as polished.  

What to Do

Sometimes the question isn't how to do something, but what do to in the first place.  I'm pretty good at the How, but coming up with the What is the tough part.  then again, if it was easy I would probably be more entrepreneurial.  I'm not worrying about the Who, the Where and the When.  Those sort of get dictated by the What.

So here's my partial list of things to do.  I'm not saying I can do these things, but sometimes a bit of thought on the subject will lead to something that I can do.  This won't be the last of these - Sometimes I feel the need to expound on something different every once in a while.

World peace

So how do you bring around the end of wars, strife, conflict, and so forth?  Mankind itself has been at war with itself since, well, the beginning of mankind itself.  When you think about it, most of it comes down to competition and the need for power.  Wars are fought over numerous things: land, resources, ideology, power, discovery and immortality.  So if you cannot remove the source of the competition, you need to redirect it in some way.  

The competitive spirit can be seen in a number of different ways.  Sports and direct competition isalways a good way.  beating someone in an event is always a good way of keeping this going, whether in the buildup or the event itself.  Take the World Cup (or the Superbowl, for those of you in the states).  the game happens periodically, but the hype before the game lasts a longer time.  the competion itself may only last a few hours or weeks, and involve twns of people, but entire populations get behind their teams, for better of for worse.  

for us non-physical types, this competition can be had through more cerebral competions, whether the math olympiads back when I was in high school, to the programming contests, to the hackathons.  Competition is strong in the software industry, as can be seen in Silicon Valley emplyment.  

World hunger

World hunger is actually a relatively easy problem to solve in theory.  the world is already capable of growing enough food for everyone to be properly fed.  The problem is not in this but in the political, logistical, and idealogical areas.  This is not only food, but also just about any other resource.

Politically some governments don't want to have anything to do with another government.  This has the effect of placing artificial barriers on getting things from one location to another, or mistrust in the goods that are offered.  Some people will refuse to take what s offered gratuitiously, and others would rather throw out or destroy what they have rather than see it be used without profit.

Logistics is the process of managing and transporting goods from one location to another.  It involves not only getting things from point A to point B, but doing it in the correctt time, in the correct way, and using the least amount of resources in the transport.  For instance, Some food can be tossed in a container and dropped anywhere, while other items will need refrigeration or delicate handling so that it will not spoil.  

Finally, ideology is related to politics, but can be more ingrained.  Politics may prevent somethingfrom happening, but there is a way for the two sides to possibly work together, and still get things done, whether directly with a treaty, or obliquely through some third party, allowing everyone to save face.  an idealogical barrier will prevent this from happening, since it is part of the parties inner core to hamper or halt this, due to differences.  

Where do lost socks and pens go

Yes, some things will border on the silly.  but think about it.  you toss your things in the dirty clothes hamper, dump them into the washer, followed by the dryer, and as you put things away, you realize there is one lone sock sitting there.  you don't remember dropping only one in there, but it must of happened,  sometimes you toss it into the sock drawer by itself, sometimes you put it someplacce special, but invariably it remains alone for the longest time.  I have had this happen multiple times, in different situations.  I have gone through my entire sock drawer, matching up everything two by two, and come up with extras.  I buy socks six pairs at a time, and after a while attrition kicks and and I cannot match things up.  Now sometimes I do toss a sock out, whether because it has worn out, or gotten stained, but that doesn't always account for all the missing foot covering.  So where do they go? I don't know.

Pens are the same way.  you grab a pen, and they invariably get set down someplace, never to be found again.  Now I know where some pens go - they stop working, so I toss them.  Some I hand to other people, since they look like the y need a pen more than I do, and some get used for other purposes.  but there are others that I put in specific places, such as in coat pockets, the car, satchels, and the like, that mysteriously disappear over time.  It is probably due to one of the reasons listed above, but I like to imagine, like Douglas Adams, that they have transported themselves through some pen shaped wormhole to a planet suited for a pen based lifeform, where they do pen-related activities and live the pen equivalent of the good life.

Now I know where hotel pens go to.  I've got a lifetime supply that I have picked up over the years.  I use them as decoys so I don't lose my good pens.  

Clearing the list

I've got a number of tasks I've had on my list of "stuff to do" for quite a while.  some of these are simple, such as correct the notation in a couple of tunes i transcribed, while some are more complex and along the lines of create a better way to filter and categorize email.  So i'm trying to knock some of those out right now.  

The music one isn't too hard.  basically I've got a book of music that I've been working off and on for about 10 years, and every once in a while it needs to be updated, or someone finds an error.  In this case it was an error in a tune that needed to be corrected.  To do this it is a multi-step process, the first of which is to make sure I have all of the software and the source material.

First off I fire up Finale and open the tune source in there.  years ago I notated everything in an earlier version of Finale, so loading wasn't too much of a problem.  I found out that it had been a while since I last updated things, since the first complaint I had was it needed to update to the new format.  I think I originally had things in Finale 2003, and now I had Finale 2008, so there are minor differences.  I should get the latest version, but probably won't for a while - this version does what I need it to do.

Finale is a software package for notating and writing sheet music, and has quite a number of features that make it stand out from those various free packages.  It does have a couple of drawbacks, though.  the first is it takes a few steps to get it into a book format, since it is geared towards writing orchestral scores and part sheets for a single piece of music at a time, rather than a book of distinct pieces.  

So I made my changes (which involved removing and redefining some section marks.  I saved the results, and started step 2.

Step 2 is getting it into a document capable of creating pages, indexes, text, and images.  To do this  I went with Microsoft Word.  My version of this software has also changed, so there are some differences in formatting involved here and there.  to get the music into the word document I first print it off as a PDF or XPS file, expand it out to a common size, and then do a screen print, pasting the results into the word document.  Yes, it is a bit convoluted, and i lose some resolution here and there, but it works.  

After pasting in the new image, I realized that it looked different.  In the past 10 years the resolution has gone up tremendously, but I haven't kept the document up since about 2006.  I'lll need to go through the book a tune at a time and correct everything.  I'll also look at a better way of embedding the images into the document so that they look better.

the other thing I was doing this week involved spam classification.  I (like most others) get a lot of unsolicited email, commonly known as spam.  I have a spam filter, but it isn't the greatest.  I've tried a few others, but in some cases they fall short.  I'd like a filter to be able be more accurate, and possibly categorize the mail even more.  Put the Faire related stuff into one folder, the Need to Keep stuff in another, the spam into the spam folder, and the Nigerian scam emails into it's own special place.  So I am playing with this to better hone some skills in design and testing, and investigate techniques for categorization  and detection of spam.

So at the first cut of things there are a handful of boxes on the sheet.  First is something to work with the email server.  second is the classification engine, and third is something to tie the two together.  I'll need some more stuff, but those are the first cut components.

so first, the mail library.

I didn't want to write the raw connectivity to the mail server.  IMAP and POP3 are well defined, complex, and I didn't feel the need to do that part of things.  A couple of Google searches later I found a number of dot net libraries that I could make use of.  After writing some test applications and seeing how they behaved, I finally settled on MailKit.

http://jeffreystedfast.blogspot.com/2014/02/introducing-mailkit-cross-platform-net.html

https://www.nuget.org/packages/MailKit/

This library was relatively simple to use and had everything I needed.  after making sure that I could easily navigate through my email and view individual messages I added the NuGet reference to my project solution and continued on.

One thing I wanted to do was to be able to run this as a separate process from the mail client itself, which would allow it to be used for a wider array of clients.  there are two ways to get email from a server:  POP3 and IMAP.  POP3 doesn't leave the email on the server, while IMAP does.  IMAP also gives you a number of other capabilities, such as folders, and the capability to run multiple clients at the same time.  While I have seen separate filter applications for POP3, they need some specialized setup on the client to read their email through the filter application, and the application also has to act as a pseudo email server.  It also makes it very hard, if not impossible, to secure the channel because the filter needs to sit in the middle of the connection, which it can't if you want it to be secure.

IMAP, on the other hand will allow multiple clients accessing the server at the same time, each on its own connection.  this would allow the filter application to be a separate client and manipulate the email separately, which would then be reflected on the actual email client.  

So I had my email connection.  I created some scaffolding around it to make it do what I wanted it to, and went to the other side of the software.

To filter software you need something to do the actual filtering.  there are a number of different things that you can do:  whitelists, blacklists,  SpamHaus lists, and text analysis, to name a few.  a good spam filter will usually make use of another of different things to better do things.  

A whitelist is a list of email addresses (usually) that you will unconditionally accept as being valid emails.  while it can be other things than an email address, those are usually the main thing that is accepted.  I will be implementing a whitelist based on email addresses.

a blacklist is the exact opposite.  It is the list of email addresses that you will immediately toss out as being spam. These are usually a bit harder to manage, since spam will not always be consistently coming from the same people all the time.

SpamHaus maintains a list of IP addresses and domain names for known spamming operations.  You can connect to their servers and check the senders IP and domain to see if it comes from one of these locations.

http://www.spamhaus.org/

textual analysis is the process of looking at the text and determining whether it is spam or not based on the text.  usually there will be a set of rules and algorithms for this process.  I'lll be exploring a number of different methods down the road, but the first will be a naive bayesian analysis of the words in the message.

the Reverend Thomas Bayes developed his theorem on probability well before the US was a country.  basically it allows you to calculate the probability of an event happening based on the total observed population.  for our spam filter the probability of an email being spam is based on the probability of each individual token being in a spam email, and then combining the probabilities together based on the total occurrence of the tokens in the entire set.

To do this I need to be able to tokenize the message, and then compare these tokens to a database of words and their probabilities.  once I have the probabilities I need to be able to calculate the probability that this email is spam, and then act on it accordingly.  finally, once the user has acknowledged that the email has been categorized correctly (or incorrectly), then I need to feed this information back into the system so that it can learn.  this feedback loop is necessary, since me guessing what words are spam related and what words are not is not going to happen.

I got the analyst working this afternoon.  it wasn't too hard - I created an in-memory repository to hold the data, and made sure I could serialize it in and out easily.  the actual math isn't hard - basically it is the sum of the probabilities of the message being spam (we'll call that P) divided by P plus the sum of the probabilities of the email not being spam (we'll call the last part NP).  so P / (N+NP).  

the database is relatively simple - it is a list of words, the number of times they have occurred in an email. and the number of times they have occurred in a spam email.  the training function is also relatively simple.  after the user acknowledges that the email is categorized correctly, I can then update the appropriate row in the table - increment the number of occurrences, and if it is a spam message, the spam count.  

the final part will be developing the interactions with the mail server.  I have the capability to watch the mail folders for changes.  when a change occurs my application will be notified.  here's what I see as happening:

when a new message arrives, categorize it.  if it is spam, move it to the spam folder, and if it is good, leave it in the inbox.

if the user deletes the email from the inbox, it is validated as good.  update the database by incrementing the appropriate word counts.

if the user deletes the email from the spam folder, it is also validated as spam.  update the database.

if the user moves the email to the spam folder, or vice versa, then we had an error in validation.  again, we update the database (or we can wait until the delete time again - that might be easier)

this can also be extrapolated out to multiple categories.  we do the same process, except there are more folders to monitor.  when an email comes into the inbox, we again categorize it, but instead of being a single percentage we do the same process for each category we are tracking.  the category with the highest percentage will win.  moving the emails will again trigger the miscategorization action, and deleting will trigger the validation action.

While doing this I determined that I will need a couple of other components, primarily dealing with the UI and running the application itself.  I'll need a Windows service to run the actual spam filter, and a UI to allow the user to manipulate the parameters for the system a bit.  I also might need another small UI to enable better testing by allowing me to run the filter outside of a service, since services are annoying, to say the least, to run in a debugger.

So next week I'll get my middle layer in place - the one tying the mail components to the analysis components, and I'll have a fully functioning spam filter.  

And the music reedit - that will be needing done some time or another, too.

Monday

Wake up.

Drop daughter at school for holiday camp.

Have breakfast.

Do dishes.

Stretch and exercise a bit.

Eat lunch.

Take some stuff to Goodwill.

Teeth cleaning at dental hygienists school.

Pick up daughter.

Get some groceries.

Cook some groceries.

Eat dinner.

Put daughter to bed.

Watch the evening news.

Help Wanted

Every once in a while I like to look T the job market, and see what is going on.  I recommend this for everyone, but don't make it an obsession.

Now there are a lot of sites and apps that make this much easier than browsing the local paper and circling likely candidates with a pen, which I distinctly remember doing in 1987 and 1988.  I've been through the headhunter/recruitment firms, I've talked to people directly.  I don't flip companies much.  Once I find a place I'm usually inclined to stay there for a while, unless it isn't a good fit.

So I'm looking out there.  Basic search parameters are something as a Microsoft developer/architect, and something very close.  Believe it or not, there is a lot within 10-15 miles of me.  Richardson used to have a strong telecom presence, and Plano and Frisco both have a number of places along the tollway.  But one place stuck out to me.  It is a small firm that I had never heard of, and the location was just down the street from me.  So I sent them my resumé.  

My resumé is not short.  While it lists only two companies I have worked for, it is over four pages in length.  I've done a lot in the past fifteen years, and the standard "designed, developed, and tested software in a Microsoft environment for multiple clients" just doesn't do it justice.  It's long enough that your typical recruiter will weigh it and toss it because anything that size has got to be padded.  But mine isn't. Each of the projects mentioned there is a story.  Each has different facets in what I do, what I am, and how I do things.  

I included a short cover letter telling who I was, and I was on a sabbatical.  And it got their attention.  I actually got a call the next day from the company president, and agreed to talk to them the next day.  Him and I spent a pleasant hour sizing each other up, answering each other's questions, and finding out what got each other's attention.  I got there attention with an honest cover letter and a resumé, they got mine by being down the street and being a viable company.   

Before going any further with this firm they wish to make sure that they hire the best, and they believe in using a standardized test for a baseline.  While in some respects this type of testing doesn't really tell the whole story, accompanied by other things it will give a good baseline to measure candidates by.  I'll take the tests this week and see how I do.  

So back to the sites that you can look at for jobs.  There are quite a few, and that is not even counting the direct company sites.  They are populated with more head hunters than a South American jungle, and there seems to be quite a lot of insurance places looking for sales people.  A lot of these places also have corresponding apps.  I'm not familiar with the Android apps, so you'll have to hunt those down on their own

Place to look if you're looking

Dice.com

Monster.com

CareerBuilder.com

TheLadders.com

GlassDoor.com

Indeed.com

And others can be found at 

http://en.wikipedia.org/wiki/List_of_employment_websites

iOS apps

Monster

Glass door

Career builder 

Job aware

And if you know of more, send them over. 

Exercise

As I lay panting on the floor, I have come to realize that I am well out of shape.  I'm fifty, not too much overweight, and most of my medical measurements are within normal parameters.  but physically I need help.  I tire too easily when doing any sort of working out, I work in a profession that is almost all desk work, and the most exercise that I get is when I walk to the restroom.  I need to change that.

I've been going to a physical therapist for the past couple of weeks to get my back into shape, and that is working.  a forced hour every couple of days is doing some good, and I started wondering, "can I keep that up?" or even expand on it a bit.  maybe not an hour, but something more than plopping on the couch at the end of the day.

I've tried this before.  a Bowflex, free weights, a treadmill, and other stuff have come and gone in the house, but to no avail.  the Bowflex at least had the advantage of being able to hang clothing from it for a time.  I've downloaded apps to my phone and tried them, but usually they get ignored after a while.

Over the weekend I did go biking a couple of times.  Now when you think biking, you might be thinking those thin guys in the skin tight suits that you see in packs and riding bikes that cost roughly the same as your first car. Well, that wasn't me.  For these two treks it was on a trail bike with 26" tires, and the speed was slightly faster than a walking pace.  And ice cream was involved.

My daughter is six.  I've been threatening to rip the training wheels off of her bike for months, and even did so in November, but put them back again.  On Saturday I pulled one of them off, and tower her if we could make it to Culver's, I'd buy ice cream.  There was wailing and gnashing of teeth, but we soon set off for the mile and a half trip.  I carried three things: my phone/wallet, a two for one coupon, and an adjustable wrench.

If you can remember back to your first trip or two without training wheels, it is a harrowing thing.  You tip left and right, and finally land in the grass.  It is the same with only one wheel, except you only have to worry about crashing one direction.  There was crying involved, but there was still ice cream at the end.  So we persevered.  

One think I realized is the remaining trading wheel was pushing the bike the other way, which made it hard to balance.  I was able to adjust this quickly by pulling the training wheel up and out, meaning that if she wanted to use the training wheel she'd have to lean into it a bit.  So she did. But there were periods where the training wheel wasn't in use.

Culver's is a burger and ice cream joint a short distance from us.  Their claim to fame is butterburgers, but their frozen custard comes in a close second.  For comparison, imagine a bowl of the best from Dairy Queen.  Now make it denser, with better flavor, and have, in addition to real vanilla and good chocolate, a flavor of the day.  Saturday's flavor was chocolate covered strawberry, which rates up there quite a ways for me.  We skipped the burgers and cheese curds this time around, though. 

Sunday's trek wasn't as far, just up to the park and back.  The cold front was starting to come in, prefaced by the freshening wind, and the rain not much behind it.  We made it back before things got nasty, though. 

In a couple of weeks I'll remove the other training wheel, or bend it up so far that is isn't useful.  If all goes well then she'll be on two wheels after that, and maybe get a larger bike for her birthday.

I'll keep doing the physical exercise each day.  I have started this two weeks ago with the physical therapy, and will continue to do that along with more strenuous stuff as I build up to it.  

Free Reeds

The last couple of days have been spent in doing a myriad of things that needed to be done around the house.  Cleaning light fixtures, taking stuff to goodwill, washing laundry and dishes (in the wash machine and dishwasher respectively) and cleaning up after the cat throwing up repeatedly.  I've still got a couple of rooms that need some serious demessing, but th house is about to the point where I can just put away the things there out of place.

I spent some time with one of my upright basses this morning.  My Merchant bass had a problem with the neck that was fixed a few years back, and I was afraid it was coming back again.  Fortunately it isn't a problem and should be good for a few more years before I need to take it into the shop.  That means I have two accordions, a couple of flutes, and a balalaika to repair.  I've got a spare folding table that I can turn into a workbench for this purpose, as long as I can keep my daughter off of it while I'm doing this.  I'll probably tackle the newer accordion first, since shifting the blocks will be simple.

I haven't really played accordion since I was around ten, when I got one at a garage sale.  I played for a while, learning the odd polka or two, but I'm no Al Yankovick.  That one disappeared into the annals of history during my college years.  It wasn't until I picked up a concertina that I started getting back into the free reeds again.  Now the are some differences in how they are played, though.

First off there are the obvious differences.  The accordion has a bunch of buttons on the left, and a keyboard on the right.  Usually you play the melody on the right, and the chords on the left.  And another important thing - when you push or pull the accordion you get the same notes.

The Anglo concertina (there are different kinds) can have anywhere from twenty to forty buttons, but usually thirty.  These are split between the two sides of the concertina, and like the accordion there is a bellows in the middle.  But unlike the accordion, you will get different notes depending on whether you are pushing or pulling on the concertina.  So not only do you have to worry about hitting the correct button, but you have to be moving the bellows in the correct direction.

Another difference is how things are laid out. On the accordion the right hand is pretty much like a piano keyboard.  From top to bottom (if you're strapped in) the notes go up in half and whole steps.  The left hand is a bit more complex, but is usually in fifths.  The home row is second from the front, the first row is the third up, and the third through sixth rows are major, minor, dominant seventh and diminished seventh chords.  And there's a bunch of them.  Usually 120 buttons, giving you twenty notes and their chords, enough to cover the entire spectrum of the keyboard and then some.  Usually there's some repeating to allow someone to hit a note in a couple of different positions. 

The concertina, well, isn't quite as neatly laid out, though there are some patterns that make some sense after a while.  The home row (usually the center row, or second away from you) will be the lower notes, while the row nearest you will be the notes a fifth up.  These are repeated on both hands, with the right hand being an octave up.  The pull will give you the minor a whole step up.  So for a D/G concertina, you get the low G major, low A minor, D major and E minor chordings.  These are very common in Irish/Scottish music.  

But, this only covers the buttons nearest the top.  Once you start getting into pinky territory the rules change and can be different from player to player, depending on where you learned.  The third row gets even worse, since there isn't too much of a pattern to it.  Actually the pattern in it is to supply the notes that the other two rows don't have, so that you get a fully chromatic instrument.  A good way to think of how a concertina operates is take a harmonica and split it in half.  Like a harmonica you get different notes whether you breathe in or out, and some of those notes can get a bit wonky depending on where you are at.  

And after that it gets even worse.  So far I have been going on about the Anglo concertina.  there are a number of different kinds, such as the English concertina, the bandoleon, the Melodeon, Harmoniums (which are pumped with foot pedals like an organ), as well as different shapes, sizes makes and models.  

Both instruments are strapped on in some form or fashion.  The accordion is like a backpack on your front, while the concertina is more of a set of handcuffs.  The accordion also weighs quite a bit more, though some of your larger concertinas can be weighty after wielding it for three hours on stage.  

I've never played an accordion professionally, but I heard that they are a lot easier to amplify than a concertina.  because of the size there is a lot of space to drop a microphone inside, or even multiples, and get a full sound out of them, much like amplifying an acoustic guitar.  the concertina is a bit harder, since space is at a premium.  my current rig is an AudioVox two microphone system that puts one mike on each side, velcro'd on close to the straps.  those run to a box on my belt, and from there a cable (or a wireless pickup) to the sound board.  I have used multiple microphones on stands, as well as another portable system, but this one works the best for me.  while the stand system works, it allows no movement at all, and unlike a singer you have to remain precisely between the two microphones, since the sound from a concertina can be very directional to the sides.

I have owned three concertinas in the past twenty years, two of which I still have.  The first was an old Hohner I picked up at a flea market.  This is one of the mass marketed models with the red end caps, paperboard bellows, and generally good for not much more than learning.  It was, however, cheap, at around $40 at the time.  It worked well for a few years, though after a while it was held together with duck tape and warm thoughts. I gave it away when I left Michigan, since I already had another one.

Hobgoblin had a model called the Goblin, which was a thirty button D/G.  I picked it up at a celtic festival in Chicago for around $500, and it has traveled with my for quite a while.  I played it in Michigan as well as Texas, and it has served me well.  I still play it every once in a while, since the sound is mellow and goes well with other instruments.  the only drawback I have had with it is the end caps are wood, which over time has become worn, allowing the buttons to slip a bit, and sometimes getting stuck.  I know a lot of people that have seeen me play this instrument at a show will remember me getting a nasty look on my face, and grabbing a screwdriver, jam it into the end piece to push the button back into place.

my last instrument I have had since 2004, and it is made by a local concertina maker in the Dallas area by the name of Harold Herrington.  I think he is retired now, but someone has taken over his business and is continuing the trade.  the Herrington is a smaller bodied concertina, but heavy.  Because it was a custom instrument I was able to get the notes as I liked, keeping the D/G fingering.  He uses steel reeds, which make for a louder tone, but they also don't go out of tune or get affected by temperature like brass does.  The metal plates on the end, with a plastic alignment plate underneath keeps the problem with button slippage at bay, and wider straps make it more comfortable to hold.  This one was $1500 back when I got it, which is comparable to other instruments in the range.  I probably won't get another, since past that point in terms of price you are primarily getting into nuances of styling, or antique models.

Nap time

Sometimes the best way to attack a problem is to ignore it for a while. Sneak up on it from a different angle.  Heck, take a nap and let your subconscious mull it over for a while.  

And sometimes you don't have a problem, but the nap is still welcome.  

Today was such a day.

I spent some time arranging and putting away things upstairs.  Legos, hot wheels, trains, a blanket fort, a doll house, and a bunch of Christmas decorations were moved more or less to where they belonged, or at least closer to where they should be.  We now have floor space where there was once a quagmire of toys.

Actually I did have a conundrum to work my way through.  I'm doing some learning on neural networks and was running into a problem on how to represent nodes in the best way.  I hacked together a couple of things this morning, but wasn't pleased with the results.  It was during my mulling over period (the aforementioned nap on the couch) that I devised a good way to do this.  So remember, kids - that person you see at their desk with their eyes closed may not be resting, but solving the worlds problems.  Don't disturb them.

Today's dinner was leftovers.  Over the course of the week we normally accumulate dibs and drabs of stuff that are not large enough to prepare for a meal, but if you add them all together you can get a meal out of them.