The Minimalists (+3)

Three days in. "The Minimalists" approach is working just fine. Not a lot of revelations about the must/should list. I went over the must list and removed some "should"s. Given my recent move there is not much to throw away. Goodwill is going to have a Christmas party this weekend with some of my poorly fitting clothes.

The Minimalists

I ran into The Minimalists web site/blog by accident this morning and was pleasantly surprised by the depth and immediate applicability of the ideas. I think these guys are onto something!

My friends know my views on a lot of these issues, as well as how the distractions and temptations along the way. Emails (71 on Sunday after filters vs. 160 on Friday), Christmas euphoria, Black Friday, etc.

Time to make a change - I am on Day 1 (Deciding). Will keep you posted on how it goes.

Magsafe2

I just got my new Macbook Air. The old one had some battery issue, after just about over a year of service. I have to say that I love my Air. I think it is the best laptop (or whatever category it falls under) ever made. The one disappointing thing is that the new Air has a new power adaptor connector - something called "Magsafe 2". I am sure someone at Apple had a reason to change it. My problem is that I have at least 4 or 5 legacy Magsafe power adapters - one in each office, a couple at home, one in my laptop bag, etc. At $80 a pop, that's another $400, in addition to the cost of the laptop, that I need to invest, just because the battery of my old Macbook Air died. Now I have to carry the $10 "adapter thingy" in my backpack, lose it, buy another one and most importantly - run out of power when I needed. Sounds like a bad idea altogether, and other people seem to agree:

http://pogue.blogs.nytimes.com/2012/07/30/one-of-apples-best-ideas-ever-made-worse/

Falling Forward

With my deepest gratitude to the friends that inspired this conversation.

We all seem to fail. Every single day. Think about it – if we can succeed 51% of the time we'd be either win the lottery or play the stock market. Failure is just an integral part of our existence – something that we cannot pretend does not happen to us or around us. Although this seems like a certain fact, I am going to create a distinction between falling and failing and will argue that the only reason for failure is the lack of commitment, not the environment,

In a conversation with someone about creating a possibility for us being an extraordinary team – a team where people actually want to be part of, despite all what is involved – I sensed that my friend was losing it. He was not enrolled yet, he did not get the possibility and did not get get it that I am committed to him and to this possibility. As my impatient self was already wondering "what is so difficult go get?", it suddenly hit me – he was afraid of failure. I just saw it right in his eyes. It was one of those very rare 'make it or break it' moments. Thought about it for a second and said - "Look, you know how much I respect you and enjoy working together. Our team is a critical juncture with a lot of responsibility, and will take a lot to turn things around. You need to make a choice – you are either on board, or not. Either way, I will support you though and through. If you're not, I'll help you get a new job – you're talented and smart, that is not an issue. If you are – I'll be there to support you every step of the way. The only thing I can't let you do is silently fail us by not making a choice.". My friend thought about it for a second as well, and replied. "I am on board. Appreciate the opportunity. Will let you know if I ever change my mind.". He got it, right there and then, that I am taking a stand for him in the matter and I am not going to let him fail. Fast forward several months ago,  yesterday as I was having an enrollment conversation over a glass or two of wine with someone else who is very dear to my heart, I was replaying this conversation in my head trying to choose my words and realized that "failure" is a tricky word: what most people call "failure" is just life. Sometimes we could have predicted it, sometimes not. Sometimes there's something to learn from it, sometimes not. If you are in a game, you stand the chance of not winning, or getting bruised. That's the nature of the game. If you play, you fall forward, get up and keep playing. You don't fail. The only way to fail is not to play. The only way to fail others is not to tell them that.

New "new" BING

I rarely express strong opinions about consumer Internet services in public, both because there's very little use to it and because I am no expert in it. This is one of the few exceptions.

I have been a big proponent of bing even after I left Microsoft. I thought they were onto something - between the design, a lot of clever HTML optimizations and semantic search capabilities I was a firm believer that bing is a better (than Google) service. I was probably one of the few bing outliers with OS X/Chrome.

It was all good until the launch of the "new" social BING. This thing is terrible. With all the popups, prompts and multiple scrolling panes I am getting dizzy. The "social" pane is jumping up and down as I am scrolling through the search results, there is so many distractions on the screen that I cannot focus on what I am really trying to accomplish. From the point of view of someone who spends quite a bit of time navigating Internet content this seems to be the "anti-search" engine. And the last thing I want is to burden my friends with my own search escapades.

Sorry Microsoft. You lost me on that one.

IPSec Performance Benchmarking (Is the end of network "hardware" near)

Introduction

After over two weeks of debugging IPSec tunneling performance issues with variety of Cisco router and firewall platforms my team decided to try IPSec tunneling on Linux. There were some impressive results published online. Initial testing between two Dell Dual Xeon servers produced marginal results of 50Mbps. After some research we realized that in order to achieve high IPSec performance we need to use the Intel AES instruction set encryption optimization. Intel's AES instruction set provides a set of CPU instructions that implement AES encryption primitives in microcode with significantly higher performance than a kernel implementation. Support for the AES interaction set has been committed in newer Linux kernels – 2.6.32 or later. Intel has a good paper describing their test bed here.

Test Setup

For our testing we decided to use the latest 2.6.35.13 and StrongWAN IPSec tunnel. Since the hosts were CentOS 5.5 I ended up compiling the kernel from source. Make sure the AES-NI driver is checked in "make menuconfig". I also compiled the latest StrongSWAN 4.6.2 from source), without any special configuration options. ./configure && make; sudo make install

The test network was very simple:

The AES acceleration requires the "aesni_intel" driver, please make sure you run sudo /sbin/modprobe aesni_intel. 

Verify that the driver is loaded:

/sbin/lsmod | grep aes

aesni_intel            11278  9 

cryptd                  6545  4 aesni_intel

Once the driver is loaded, you can start the IPSec protocol stack:

sudo /usr/local/sbin/ipsec start and bring the tunnel up sudo /usr/local/sbin/ipsec up host-host

Below is the ipsec.conf for both hosts. Do not forget to configure ipsec.secrets as well (documentation is available on the StrongSWAN web site). The ESP encryption we tested was 128-bit AES-CCM-8 (as the original Intel paper) I expect that your mileage may vary, depending on whether specific cipher suite is supported by the AES driver or not. 

After initial negotiation the tunnel came up:

-bash-3.2$ sudo /usr/local/sbin/ipsec statusall

000 Status of IKEv1 pluto daemon (strongSwan 4.6.2):

000 interface lo/lo ::1:500

000 interface lo/lo 127.0.0.1:500

000 interface eth0/eth0 10.0.0.1:500

000 %myid = '%any'

000 loaded plugins: sha1 sha2 md5 aes des hmac gmp random kernel-netlink

000 debug options: raw+crypt+parsing+emitting+control+lifecycle+kernel+dns+natt+oppo+controlmore

000 

000 "host-host": 10.0.0.1[moon]…10.0.0.2[sun]; erouted; eroute owner: #2

000 "host-host":   ike_life: 3600s; ipsec_life: 1200s; rekey_margin: 180s; rekey_fuzz: 100%; keyingtries: 1

000 "host-host":   policy: PSK+ENCRYPT+TUNNEL+PFS; prio: 32,32; interface: eth0; 

000 "host-host":   newest ISAKMP SA: #1; newest IPsec SA: #2; 

000 "host-host":   IKE proposal: AES_CBC_128/HMAC_SHA1/MODP_2048

000 "host-host":   ESP proposal: AES_CCM_8_128/AUTH_NONE/<Phase1>

Note the ESP proposal:

For testing we tried several different methods – ssh, iperf and netcat. The netcat test setup is the easiest to replicate. To test unidirectional traffic flow, open two ssh sessions to each host:

On moon start:

Sesion1: nc -l 1234 > /dev/null

Sesion2: nc -l 1235 > /dev/null

On sun:

Session1: dd if=/dev/zero bs=1M count=10000 | nc moon 1234

Session2: dd if=/dev/zero bs=1M count=10000 | nc moon 1235

To test bidirectional traffic flow performance try:

On moon start:

Sesion1: nc -l 1234 > /dev/null

Session2: dd if=/dev/zero bs=1M count=10000 | nc sun 1234

On sun:

Session1: dd if=/dev/zero bs=1M count=10000 | nc moon 1234

Sesion2: nc -l 1234 > /dev/null

Host moon:

/usr/local/etc/ipsec.conf:

config setup

   plutodebug=control

   charonstart=no

conn %default

   ikelifetime=60m

   keylife=20m

   rekeymargin=3m

   keyingtries=1

   keyexchange=ikev1

   authby=secret

   esp=aes128ccm8-modp1024

conn host-host

   left=10.0.0.1

   leftid=@moon

   leftfirewall=yes

   right=10.0.0.2

   rightid=@sun

   auto=add

Host sun:

/usr/local/etc/ipsec.conf

config setup

   plutodebug=control

   charonstart=no

conn %default

   ikelifetime=60m

   keylife=20m

   rekeymargin=3m

   keyingtries=1

   keyexchange=ikev1

   authby=secret

   esp=aes128ccm8-modp1024

conn host-host

   left=10.0.0.2

   leftid=@sun

   leftfirewall=yes

   right=10.0.0.1

   rightid=@moon

   auto=add

Test results:

Single stream:

dd if=/dev/zero bs=1M count=10000 | nc moon 1234

10000+0 records in

10000+0 records out

10485760000 bytes (10 GB) copied, 134.506 seconds, 78.0 MB/s 

624Mbps unencrypted traffic

Two parallel streams (unidirectional):

8814329856 bytes (8.8 GB) copied, 288.863 seconds, 42.5 MB/s

8814329856 bytes (8.8 GB) copied, 288.863 seconds, 43.5 MB/s

688Mbps unencrypted traffic

Bidirectional traffic tests yielded similar results - each stream was able to push around 350-400Mbps for a total of 800Mbps unencrypted traffic. Including the IPSec overhead this pretty much maxes out the Gigabit Ethernet interface on the host, the throughput of the NIC maxes out at 900Mbps (encrypted traffic). The encryption seems to consume 1 2.4GHz core (99% utilization) for encryption/decryption of 800Mbps traffic (either uni- or bi-directional traffic). The rest of the cores were pretty ugh idle. We did not try anything to spread the load between multiple cores and get higher throughput because for our use case this is sufficient amount of bandwidth.

Why did I decide to write about this?

I think this is important for two reasons: one, it is amazing how much the performance of  "commodity" CPUs have increased, even for highly specialized tasks like AES encryption. It seems that Intel is going after a variety of market niches, which used to be served by a variety of manufacturers like Cavium. Two, a lot of solutions which even 3-4 years ago were possible only using dedicated "network hardware" - routers, firewalls, encryptors, etc. now are possible at much lower cost, greater flexibility and "no strings attached" on off-the-shelf Intel Xeon servers running Linux.  The Open Source software (like StrongSWAN) is becoming much more advanced and easier to administer, the availability of source code allows the more advanced users to modify and fix it as they see fit.For comparison a similarly sized Cisco ASA (ASA 5585-X with SSP10 , capable of 1Gbps encrypted traffic) runs $40k/each, with SmartNet support costs that add up over the years. Additionally, hardware replacement is not as easy, configuration requires specialized knowledge. At this point I don't see a lot of rational reasons why anyone would purchase a $40k firewall (add to that you need two per site + support contract), instead of just running StrongSWAN on two Dell PowerEdge servers. That is $150k savings, give or take – absolutely worth the day's worth of work to set it up and test it. I am sure that a lot of enterprises will reason why they should continue spending money on "dedicated network hardware" but they are very much the same arguments they had 10 years ago why they should keep buying underpowered and overpriced "commercial Unix" platforms like Data General, DEC, HP-UX and AIX. 10 years after resellers are GIVING AWAY HP-UX servers for the price of two-year support agreement.

Between the commoditization of Ethernet switching by Marvell, Broadcom, Fulcrum/Intel, advancement of Open Source software, OpenFlow/OVS, provisioning and orchestration like OpenStack, there is a rapidly growing set of alternatives to monolithic network "hardware". If I were Cisco I'd be worried about the switch and firewall sales.

Nagios to ThousandEyes Integration

The company I work for (ServiceNow) recently started using a SaaS for external monitoring from a startup company called ThousandEyes and we decided to standardize on it. The guys from ThousandEyes have looked into some use cases which other players did not address, particularly for troubleshooting routing anomalies in real time. I also liked the UI very much – it is clean, responsive and can be used both for ad-hoc queries and for "NOC" dashboard. Although the service lacks some features that the competition already has we decided to adopt it because of the great troubleshooting features, ability to view network changes and the agility of the company. One issue that the NOC team brought up is they would like to look at alerts in one place – the Nagios console, not have to look at both Nagios and ThousandEyes, so I wrote a small integration between ThousandEyes REST API and Nagios.  I have not had the time to put it on github, so if you're interested - please email me directly.

Smart phones and dumb phones

I was sitting on the plane on my way back from San Diego reading my email on an iPhone, the guy next to me was doing the same thing on a Blackberry. We were both passively asocial and didn't care much about socializing, as I watched him with my peripheral vision typing on the keyboard I remembered the days when Blackberry was the ultimate device – a phone, PDA, you can read your email and maybe look at a web page or stock quotes. I vividly remember the first Blackberry I had – it did not even have a speaker, required a headset but it was totally worth the hassle – I could get my email and contacts on it. A real "smart" phone, helps you stay connected and get some work done even on the road. Fast forward 5–6 years, the Blackberry can still do the same things but not much more. Can't play Pandora, there's no Angry Birds, doesn't have touch screen, the screen resolution makes you squint. Next to an iPhone (or WinMo7 or Android phone) the Blackberry looks antiquated, outdated, something from the past century. It suddenly turned into a "dumb" phone. 

What happened over the past several years? Research In Motion (the markers of Blackberry) just fell asleep at the wheel. They stopped innovating and resorted to mediocracy and incremental improvement of their product; in the meantime, Apple, Google and Microsoft leapfrogged them, both as product and business model. RIM never really believed that third-party developers can provide value and did not launch an app store. Deploying Blackberries for enterprise required software (Blackberry Enterprise Server) and a lot of configuration work. What a nightmare! It takes about 5 minutes to configure your iPhone or WinMo7 to synchronize with your corporate email (or personal) email and does not require anyone to do anything. The same thing (almost) happened to Microsoft – I personally like WinMo7 very much because of it's usability and innovative UI. It is also very solid platform with great developer ecosystem (don't want to stir up a debate on this one). It was just several years too late, Microsoft stopped innovating after WinMo6.5, which had serious usability, stability and battery life issues, not to mention complete absence of 3rd party applications. The same thing happened to Nokia, it just took a little longer – 20 years ago Nokia ruled the mobile market, today one cannot find a Nokia phone anywhere in the US except in the "old electronics" drawers. 

What I am making out of this is that mediocracy, complacency and averageness is no longer an option for a company, if it wants to last more than one product lifecycle (a.k.a. "one-trick pony"). With product life cycles becoming shorter and shorter, much faster adoption and zero switching costs there is absolutely no time to relax and bug fix a product. Consumers are becoming more and more willing to adopt "early" releases, not get upset over incomplete functionality, minor bugs, defects and outages, which makes it even easier and more compelling to take market share from the incumbent players. 

If I were an incumbent player I would be seriously worried both about my customers and my talent – most good engineers thrive on building new products, not on spending weeks fixing minor bugs or releasing incremental features. It gets depressing to work for a company that is coasting on the laurels of their past achievements. One thing for sure is that I don't want to work there, nor are the people I want to work with. I have been fortunate to work for and with some exceptional individuals that defeat common sense – they "made it" at one point or another and don't have to worry about working ever again but they still work harder than 99% of the rest. Why is that – they just don't accept mediocracy as an option. Such people are also very polarizing – they have contagious, motivating and exhilarating impact on some, while others think they are just being difficult and demanding. Both sides are right – achieving great results is both difficult and inspiring. To bring this back to the Blackberry conversation – if I were in a business which is open to global competition I will be hiring some of the "difficult" people as quickly as I can. You can easily tell who they are.

Hats off to wikipedia

to my surprise i just realized that wikipedia is off the air in protest of SOPA and PIPA. in a microsecond it reminded me how much can be accomplished by true freedom of information and how little does it take for it to go away. i will miss wikipedia for a day but hopefully will remember how quickly the dark ages can come back for a long time.