Thursday, March 24, 2011

Bandwidth crisis?

The number of paid users is growing steadily even before we do a serious marketing campaign. This, obviously, is something that is getting us very excited at the possibility of running  a successful business of selling Internet access in Mt. View estate. Residents clearly need internet service in their homes as intimated by an assistant at the Resident's association office. Here is what she had to say: 

"Residents always come here to ask about who we have decided to invite to provide internet services in the estate. We have been trying to reach out to a specific ISP but there is no positive response. We hope your (phone) number works coz the ones given by the ISP don't seem to work."

We are currently serving 13 homes, 3 of which are demo users and are willing to pay after they test the service and find it worthy paying for.Most of these users were introduced to us by our earlier customers who found the service good enough that they recommended to their friends. This is the best thing that can happen to a business since, in addition to getting new customers, you are sure the existing ones are happy. This is a marketing line we are keen to encourage.

The disturbing thing, however, has been frequent network outages we have experienced in the last couple of days which have seen even our most trusted customers complain. Here is a text I received from one of them:

" I have tried again with several machines and yet again I cannot connect. What is happening? These outages are too much. "

Most  conspicuous of the outages is the network being down every Sunday. Here is what another customer had to say:

" U have to come n check out this net 2mrw. It's been acting up lately. So so so slow n keeps on disconnecting. + every Sunday it does not work." 

We anticipated this scenario. Only it came knocking earlier than we expected. Letting down our customers is the last thing we want to do. It hurts. We must do better than the big boys with hundreds of thousands of customers to keep happy.

On the cards is emphasis on quality of service, very high uptime and excellent customer support. We are well on our way to becoming a community ISP for real.
This calls for strict adherence to bandwidth requirements. A dedicated connection fully paid for or subsidized is not a far fetched idea. More bandwidth is most welcome at this point.

Tuesday, March 8, 2011

Fabfi 4.0 Progress Report

It's been two months to the day since the beginning of my follow-up visit to the Fablab in Nairobi, and high time for a little status update, first about the networks themselves, and then about what makes them tick

The Networks

As of today, there are three Fabfi networks under development in Kenya:
  • Mountain View Estates / Kangemi
  • Njabini
  • ARO Fablab, near Bondo
All three networks are operated by local residents, and each has a unique business model.

Mountain View / Kangemi

While the latter two networks above are still pre-production, the Mt. View network has been in full production for about two months. Mt. View is the closest to US-Style commercial model where the typical user pays a subscription fee to receive unlimited access for a month at a predetermined speed.

The network currently operates with 23 nodes across 9 distribution points, providing network coverage to between 60 and 75 households (green circles are "indoor" quality coverage while purple ones are "outdoor" quality): 

After some trial-and-error with power solutions, the entire network is also now protected by 2-6 hours of expandable battery backup per node, keeping the internet available even in the event of a total blackout (which are really frequent, wouldn't ya know...)

 A few weeks ago, the network made its first leap into Kangemi District, which is a considerably more "underserved" area than Mountain View.   As the team begins to automate the monitoring and management of existing infrastructure over the next few weeks, they will be looking for locations to expand on the Kangemi side of the wall.

Of the 9 distribution points currently operating, 6 were added during the months of January and February, with word-of-mouth rapidly turning up residents eager to host new nodes.  We now have at least one subscriber for every distribution point, and John and Nick have been out and about actively recruiting more.  

An unexpected challenge in building a subscriber base has been structuring our demo program.  As you may have seen on the website, the team is offering one week free to new subscribers, however the original program of "Try it for a week, then pay"  was exploited almost immediately by groups of people calling for a user account, sharing it for a week, then calling for a new one the next week.  As a result, the offer has been changed such that your first "month" of  subscription comes with an extra week subscription bonus.

Surprisingly, we have seen very little discovery of the limited free service option.  This is the option where you can sign up and have a small amount of access per day (15min or 15MB, whichever comes first).  This option has not been aggressively marketed, but have had fewer people than expected asking about it given that it's mentioned only one click in from the main splash page. I expect we'll see a lot more of this use once users can sign themselves up from a web form on the splash page. 


As you may have read in earlier posts, I spent a significant amount of time in Njabini working with local Entrepreneur, Peter Murimi, and the Flying Kites orphanage to get a network up there.

One of the biggest challenges in his area is the mobile dongles are the only connectivity method.  In this network, the pricing structure of the mobile dongles informs a business model where Peter, the local operator can buy bulk packages from the mobile provider and share those packages among local users who buy smaller units at a price below the Safaricom rates. This is partially enabled by the use of an aggressive web-caching system that runs locally to save bits. 

While the necessity of pay-per-bit (as opposed to pay per unit of capacity) informed a solid business model, the resource constraint immediately caused us to pause and rethink our rollout strategy. In this cost scenario system overhead equates directly to cost so Tom is busy working out some proper performance monitoring software that will help us trim the fat before we go live.

The Nairobi team will be visiting Njabini this weekend to make some modifications and teach some network management, after which I expect we'll see a lot more activity out of the sit.

ARO Fablab:

On the outskirts of Bondo-town and truly off the grid, the ARO Fablab is perhaps the most extreme location for a Fabfi deploy.  Our friend Boris is out there working with them now.  Once they're consistently online the global Fabfi crew can log in and provide remote support. I received mail from the lab Manager today saying:

"Tom from Nairobi came over to assist us complete our FabFi configurations. We successfully configured the headnode and its tested and working fine. We tried to flash the pico devices with the new firmware but did not succeed so far, the old versions we did while I was in Nairobi works well.

We were planning to connect the nearby high schools and also to the cyber cafes at the nearby market centre. This would assist us in paying for the internet connection and providing readily access to the internet for school students and their teachers."

The Tech

While we like to give the impression that building fabfi consists of simply pointing a couple wifi routers at each other and calling it a day, there's an enormous amount of development that goes into creating that illusion.  The current fabfi system does first-order integration with dozens of open-source projects, including:
And with every day we're adding new tweaks to improve stability and performance that require testing in the UoN Fablab development network.


The move to version 4.0 prompted the switch to all new hardware, while all the new routers are now working very well, we're still cleaning up the prototype reflector feeds for mass consumption.  Info on the process and the designs can be found in the RF section of the wiki.

System Management

Over the last couple of months, the most development has gone into ways of keeping track of nodes and users.  We have a few things working nicely now, including a remotely hosted splash page for user logins, a live network map, and a web form that makes .kml for Google Earth or Maps showing all the important data for each node.

Tom is actively developing Nagios-based service alerting and bandwidth monitoring tools that we should be seeing in the next couple of weeks.  I also hear rumor that he wants to build a web form to let users sign themselves up in the fablab (yes, please!).


Today (fittingly at rev. 400) I finally completed the core wiki for Fabfi 4.0, and updated the fabfi site to point to the new content. We also finally have a complete site for JoinAfrica Kenya where users can learn about their local networks and get online.

The Connections

One of the most exciting parts of what's going on in Kenya right now is networking of the social variety.  When Paul came to me last spring and said "I want to find a sustainable way to get more people online in Africa," my biggest concerns were not technical (though I had a few of those) but social.  What was the use case for the internet in rural Kenya?  Who knows how to operate computers competently?  At least the latter is something that even the most technologically advanced societies on earth still wrestle with.  Sustainable internet expansion is not simply an economic issue but one of knowledge transfer.  You need to carry the knowledge of how to use the tool with the tool.
In Kenya, the UoN Fablab team is now not only expanding internet networks, but creating a network of operators and supporters that share their aims and their skills. They are actively training the operators of the two other networks, as well as working with organizations like KENET and local bandwidth providers to create meaningful partnerships.

Meanwhile operators like Peter from Njabini are setting an example of what could be the future of rural and urban-fringe networking.  By combining technical knowledge from the Fabfi team with his local knowledge of community needs and supporters, guys like Peter are key to bridging the gap between the available services and their utility to local residents. Personally, I can't wait to see the results...

Also Special Thanks To...

About a million people, but this project would especially not be possible without contributions from these major supporters and virtual team members:
  • Paul English
  • Kamau Gachigi, and the UoN Science and Technology Park
  • Amy Sun and Sherry Lassiter of FabFolk, Inc.
  • Antoine VanGelder of Afrimesh
  • MIT Center for Bits and Atoms
  • Anjali Sastry, Shiba nemat-Nasser and Nipun Virmani from MIT Sloan
  • Kerry Lynn
  • Riyaz Bachani of Wananchi Online
  • Meoli Kashorda and KENET
  • Christiaan Adams from Google Earth
  • Pete Szolovits (for letting Keith spend so much time on this project)
You all rock, and we hope you'll stay with us as we try to grow the networks described above.

Saturday, March 5, 2011

Stop Watching Me!

The wireless watchdog from the last post was short lived, as the data it sent back made a clear case for changing the wireless configuration (it also had some annoying side-effects on the link).

The link in question is firing through some foliage (if you know the guy with the trees, please ask him to cut the tops off :)), and wind seems to be having the effect of causing the signal to levels that disconnect the STA.  As a more permanent fix, we switched the link to ADHOC mode, which limits some of our HT options, but seems to fix the stability problem while still giving us link speeds near the peak speed of the uplink.  

It's funny, you can totally see the trees waving in the breeze in the ping times:

64 bytes from seq=295 ttl=64 time=1.172 ms
64 bytes from seq=296 ttl=64 time=719.076 ms
64 bytes from seq=297 ttl=64 time=1.207 ms
64 bytes from seq=298 ttl=64 time=1.805 ms
64 bytes from seq=299 ttl=64 time=425.842 ms
64 bytes from seq=300 ttl=64 time=2.308 ms
64 bytes from seq=301 ttl=64 time=6.422 ms
64 bytes from seq=302 ttl=64 time=1.321 ms
64 bytes from seq=303 ttl=64 time=2.304 ms
64 bytes from seq=304 ttl=64 time=23.272 ms
64 bytes from seq=305 ttl=64 time=1.172 ms
64 bytes from seq=306 ttl=64 time=7.758 ms
64 bytes from seq=307 ttl=64 time=21.644 ms
64 bytes from seq=308 ttl=64 time=13.355 ms
64 bytes from seq=309 ttl=64 time=9.076 ms
64 bytes from seq=310 ttl=64 time=281.356 ms
64 bytes from seq=311 ttl=64 time=1.200 ms
64 bytes from seq=312 ttl=64 time=302.242 ms

Despite the high jitter, there's no packet loss at all on the link.  Hurray for greenfield deployment!

Thanks to our new clients for their patience while we sorted this one out.  I expect that the quality of the link should be much improved going forward. 

Thursday, March 3, 2011

Watch Me!

It's been a long week of updating wiki pages and building firmware to clean up all the loose ends from January and February. A big data dump is coming soon, but to tide you over here's a little techie interlude...

As I may have mentioned before, we added infrastructure mode wireless connections to our config options to capture HT throughput (the adhoc HT driver is coming! I can practically taste it). While AP / STA can be fast, it also adds the whole complication of association. In greenfield environments, the noise floor is generally so low that we can run fast links with scary-low signal strengths (the floor on the link budget is often device sensitivity, not noise at all). In adhoc mode that works great, but in infrastructure mode, there's some setting I haven't found yet that keeps a device from reassociating with a link that's running below a certain strength (or maybe a limit on the number of reassocs, I'm not sure).

In any case, devices dropping links that are perfectly good has been an issue lately, but it seems bumping the AP's wireless connection is enough to get it back, but we can't just sit there waiting for the connection to go down?  Or can we?  Enter watchdog (details truncated for brevity):

while true
    resp=`iw dev wlan0 station dump | grep -c "[MAC you care about]"`
    if [ $resp == 0 ]
        echo `date` No assoc. bump wifi >> /tmp//log.log
        sleep 10
    sleep 5

It's not the ideal solution, but seems to be working so far.  And I'm not one to screw with what works...