Stupid sed Tricks: LDAP

Spent too many minutes doing a simple task today:

Take groups from LDAP and tell me who in in group 1 but not group 2.

 

Apache’s Directory Studio is essential if you do a lot of LDAP work. It makes it easy to navigate and peek around. With this I was able to dump two files, each listing the members of their GroupOfName records. Each line looked like:

mail=emailaddress,ou=Groups,dc=company,dc=com

All I really cared about were the email addresses. So let’s get those first:

cat userlist1.txt | sed 's/mail=\(.*\),ou=Group,dc=company,dc=com/\1/' > emails1.txt

I did that for twice, once for each file. Then I wanted to sort them:

sort email1.txt > sorted_email1.txt

Once again, twice. Once per file. I need to sort them for the comparison tool, as it expected ordered data. Finally, my in group 1 but not group 2 report:

comm -23 sorted_email1.txt sorted_email2.txt

The comm command reports three colums: only in file 1, only in file 2, and in both. The -23 switch suppresses columns 2 and 3.

 

So it took me a good 5 minutes (after 10 to remember sed syntax), simple unix tools saved me a script. And if need to be done enough, could easily be a script. Yay Unix text!

 

Oppose H.R. 3025: AKA Cellphone RoboRape

 

Dear Honorable Congressmen and Congresswomen,

Today I ask you to oppose H. R. 3035: To amend the Communications Act of 1934 to permit informational calls to mobile telephone numbers, and for other purposes.

This bill would strip citizens of the protections we have from abusive phone calls from companies. These onerous calls, which invade into a our private lives, family times, etc have long been blocked on land and cell lines.

This is a thinly veiled attempt to circumvent these long standing consumer protections. Today, many Americans depend on cell phones*. Instead of arguing that companies should have freer access to people via cellphone, I would argue the opposite. Cellphones are usually on the person at all times and thus much personal and critical.

Instead of just interrupting our dinners with Families, cellphones would allow solicitors and “robodialers” to invade our workplaces, our meetings, our commutes, our schools, public places, etc.

Many of us use our cellphones for critical contact; parents with children, doctors or professional; we live “on call.” Giving the freedom to robodialers to invade that channel of communication would greatly diminish the utility of cell phones for citizens and bring them few if any benefits in return. This only benefits corporations, which unlike citizens, do not have rights.

When your cell phone rings at 1:30 in the afternoon, as a parent you know it is an issue. When you phone rings at 9:00pm at night, as a professional you know that something important at work needs your attention. If this bill passes, you will now rush in a panic to wait through 15 seconds of silence for a recorded message inform you of your opportunity to get yet another Visa card.

I for one do not pay $79 a month for what my family calls “The Bat Line,” to subsidize yet another advertising venue.

* 91% (Arstechnica, March 2010) use cell phones, and a Reuters survey reported that 25% of Americans only use a cellphone.

Eurozone North and South

 

I was listening to an interview with the editor of Germany’s “Die Zeit” as he opined about the “Protestant North” and the “catholic Club Med” South of Europe. His contention was that the Northern countries in the Eurozone were traditionally more fiscally responsible and subsidizing the free-wheeling and spend more than you make Southern states.

The question at hand is, can they dissolve the Eurozone. Like the USA, it is a federation of independent governments that joined to create a greater economic/political power. Like the antebellum America, there is a debate about if you can freely join a union, can you freely leave it?

Lincoln realized that America would never be a serious nation and the federal government would be impotent to make tough or timely leadership if states could leave when they didn’t like what the union had decided. A permeable union would have no clout or authority, so to preserve the states and the union for those who valued it, he decided that we were compelled to keep it intact and enforce the authority of the central union over the member states.

The EU struggles because that question still lingers. Can Greece decide not to pay, can the North split off on it own. Like America’s civil war, it was more sparked by two different economies running at two different speeds roughly forced together than anything else.

Is the EU at a similar crisis point because of a similar economic situation: two different realities trying to rationalize as one. If you look at the employment and poverty rates in the American North vs. the South, you can see that there is a lot of work remaining after 100+ years. Do the relatively pampered Europeans have the stomach for what lies ahead?

Ideas evolve

A recent Smithsonian Magazine article, What Defines a Meme, gave a quick history of the concept of a Meme. The Information-Generation might like to think that they were invented along with the Internet as a way to procreate punchlines that cannot survive without context, but not so.

The idea of a Meme is older, dating back to when scientists tried to describe how information evolved and procreated like biological organisms. It is an idea that has always resonated with me. Life is nothing more that information that has found a way to procreate itself in the physical world. A meme would be information freed from a particular physical representation, but still able to procreate.

Where your genetic information needs base pairs to replicate, religion and nationalism are able to replicate at the informational level without a physical analog. I am attracted to the idea that human beings are carriers of memes the same way you could argue that we are carriers of DNA.

Without the error-checking of a physical construct, memes mutate and evolve more rapidly than genetic information. In the last century of scientific progressiveness, we often mistake evolution with progress. Evolution is actually a nasty affair of mutate and hope for the best. More times than not, life evolves to fill doomed niches, isolated corners, or the role of a disease.

In survival of the fittest, it isn’t the more correct or useful meme that survives, it is the one that is the best suited for replication between humans. Simple ideas that are easy to communicate and remember seems to be better than nuanced reason. Ideas that reinforce our prejudices are more easily injected into the host.

In this era of “I read it on the Internet,” and Fox News; I fear that our rapid communication is just allowing the most brutish of memes to spread the fastest. Just like evolution, rate of transmission and replication makes genetic data successful. There are no other criteria of worth.

In biological arms races, the fastest replicators choke out the slower organisms. They fast reproducers quickly gobble up all the resources and starve off the competition.  In keeping with a biological theme; this rapid proliferation and adaption of memes could be creating super-memes, or reason-resistant memes. In short, do our human heads have enough room for all the stupid memes? Or, like super flu-bugs, will the infection be too much?

Why You No Pattern?

Anyone who has plumbed through a Servlet container like Tomcat will recognize the “pattern” above. Not just Servlet containers, many popular Java service provider frameworks implement services the same way.

I end up explaining this “pattern” to people all the time, but I can’t find a specific pattern names to which to them. Then again, it really is about using interfaces and abstract classes properly, so it is more like a feature of the language than a true pattern. Some of the .Net pattern people reefer to something similar as the Provider Pattern.

What’s in a Name?

Unless someone can tell me the real name of the pattern, I am going to call it “Bob.” Now people will just have to know that I want the service designed like “Bob,” I mean like this.

 

How the Pattern Works

 

You have a ServiceInterface that works on a ResourceInterface. For example,

• Service Interface: Servlet
  
• Resource Interfaces: ServletRequest and ServletResponse
  

Each subclass of the ServiceInterface also subclasses the resources, so these extend in parallel. For example:

• Service: HttpServlet
  
• Resource: HttpServletRequest and HttpServletResponse
  

However, clients of the Service, typically only know about the base Service and Resources classes. Only for special clients do they care about the extended HTTP behavior and they can test the Service and Resources by instancesof.

In some cases, you may even apply multiple extension interfaces to a Service or a Resource for client to access multiple sets of extended features.

 

Where the Pattern Comes From

For Java services, the evolution starts simply from:

Interface and Implementation

• Define your service with an interface
• Provide a Basic implementation of the service that implements the interface

Interface, to Abstract, to Implementation

After you implement a few variations of your service, you find that you have a lot of common code that you want to share. Typically, a majority of the services are more similar than different. So you implement all of the common functionality in an abstract class.

You refactor your Basic implementation to extend the AbstractService.

Extend, Aggregate, or Only Implement
Now for each subsequent service you implement, you have the option to:

• Make a service that is pretty routine and simply extends the abstract service class and adds what few things are unique, OR
• Implement that interface, but because your service is so atypical, you implement your own Service from scratch, OR
• Somewhere in the middle, you implement you own service but aggregate an internal instance of the Basic Service Inmplementation to leverage some common functionality.

You can see this with the HttpServlet service. Like most services, the Servlet works as a stateless (unless you are doing something screwy) service that passes what is specific to the request in a set of messages. In the case of the HttpServlet, you get HttpServletRequests and HttpServletResponse (among other message objects).

What’s Tron with Software

Why are programs eating?!? You must acquit.

I was lucky to attend a sold out showing of the original Tron at the Santa Monica Aero following by a talk by some of the key people behind the movie.

Anyone who has seen Tron vs. Tron Legacy knows that we are talking about very different movies and only one of them really stands up as good. In listing to how the original was made, I was stuck by three very profound statements. These are truths most of us know, but I find that I seldom have a chance to reflect upon them.

Tron was made in the planning.

They spent as much time planning Tron as they did filming it. Each shot of Tron was an epic in itself. Very little of the computer world was actually done by computer. Each shot was done in film, then blown up to a full sized animation cel and copied many times. These cels were shipped to Taiwan to be handed painted. Then each shot was shipped back to America, organized, stacked, and shot as a final composite shot.

Some shots involved more than 30 individual cels! It took more than 5 semitrailers to transport all the production cels for Tron. With each shot being so expensive and time consuming, there was no room for waste.

The whole movie was shot and production finished in 13 months, about the time same time spent planning it.

Tyranny of choices

Reflecting on this, the director mused that the CGI revolution was supposed to make movies faster and cheaper to produce, but now movies take years to make and run hundreds of millions over budget. So how has this technology failed?

His answer was that the technology succeeded. Each shot is cheaper and faster, which on the micro-scale reduced the pressure to plan and execute each shot. On the macro-scale, there is less reason to plan a whole movie and to keep to a plan, instead people start shooting and say we will fix it in production. In software we say you cannot test in quality, in the same way you can post-production in a plot.

With so many options available cheaply, having lots of options replaced planning.

Subtractive vs. additive

When you had to plan a movie and had limited choices, making a movie was an additive process. Each scene is there because it took planning and hard work to to put it there. It was like building a mosaic, the entire picture exists because each tile was planned and placed.

Now many people can add their scenes and input their vision, the director spends more time subtracting things. The director carves away all the options until hopefully they have something left that resembles a good movie. But you find A plots, B plots, left over characters from the third rewrite, scene that could work if you rewrite them, someone’s pet part that doesn’t quite fit, but it is good enough.

The subtractive process tends to leave us with spaghetti movies that share a committee vision of people who were not all on the same page.

Like Code

Anyone who has worked in code sees they they live in a world of Tron or Tron Legacy. Either we realize the cost of our time and plan to use it wisely, or we just start coding with little idea of where we are trying to go. Either we use new methods to really to bring down costs and risks, or we use them as an excuse to indulge in bad habits because they make each misstep that much easier to accept. Either we are building towards good software, or we are cobbling together spaghetti code into a functionally good enough mess.

Personally, I want to fight for the user!

As Goes the Infrastructure

Data from http://gerrycanavan.wordpress.com/

The latest Monocle Magazine offered a sober view of the American decline with their article “The City Limits: New York State.”

NYC is the classic tale of a giant urban metropolis that steers a large state full of more suburban and country folk. NYC’s liberal politics and urban focus seems to infuriate that more conservative hinterlanders while NYCers complain of how their taxes and economy carry their country cousins along.

Great infrastructures are investments made by previous generations in the future continuation and prosperity of their society. Each great civilization had great infrastructure. Egypt had the Nile, Rome had roads and aqueducts, London and Paris had sewers. America is no exception, in fact we have been exceptional! We founded this nation with a post office, then laid trains (land given by government to private train companies), telephones (same as trains), post office, schools, libraries, roads, and now the Internet.

Today’s “conservative” movement has been hijacked by a lie about fiscal responsibility, that we cannot afford great social projects. In this recession, messages about responsibility resonate. However, true fiscal responsibility is about money in and money out, return on investment, and management of risks. Since WWII, the Me Generation has been withdrawing on deposits made by previous generations and paying nothing back. Today’s “conservative” movement is a ponzu game for denialist and dupes. Our nation has not been bankrupted by Obamacare, or even eight years of Bush. Individual tax payers have been increasinly asked to foot the bill of a society and are crumbling under the pressure as large corporations leverage what they can get out of our society before it crumbles.

The working class is being used like booster rocket fuel to propel the American wealthy into a global market, while they fall back into poverty.

The Monocle article shows this in the microcosm of Norther New York once prosperous cities: Buffalo, Syracuse and Rodchester. After the Erie Canal and then the rail roads, these northern cities were some of the wealthiest in the nation.  They were the home to major innovation and industry of their time: GE, Kodak, IBM. But with the decline of relevant infrastructure also came a decline in their economies. Since the 50′s, these cities have grown older, whiter, poorer, and smaller. Buffalo lost half of its population between 1950 and 2000! Kodak may have faltered, but GE and IBM reached escape velocity.

Each one of these great infrastructures allowed for explosive creation of new wealth…but it took about 20 years or a generation for the seeds to bear fruit. Now we seem to be a nation of myopic cowards, no longer looking to plant for a future beyond an immediate horizon. Fear of hard times have had people recoil from the cost of maintaining what we have let alone invest in the infrastructures that will feed the next cycles of growth. Corporate interests have demonized something that has been part of the nation since it’s founding as “socialism” and the people who don’t live in the cities rally to the cry.

We talk about fiscal responsibility, but when you look at who is paying, corporate taxes have gone down for the last 50 years while the tab has been picked up by payroll taxes. So we socialize the risk and cost of running a country, but we privatize the profits.

People working paycheck to paycheck are being squeezed and rightfully worry if they can afford this nation we live in. Meanwhile, corporations dole out large bonuses and used the saved money to build roads, telecommunication networks, and distribution channels offshore.

If we can not longer afford our roads and cities, can we afford these corporations? No great nation ever outlasted its infrastructure.

Open Letter to Washington DC

This is an open letter to our President, our congress people, and our senators:

By far, the worst thing revealed by the Wikileaks cables, has been how America has reacted. Corporations have been pressured into being secret thugs. People have been threatened with assassination by American officials. Speech and press have been shutdown.

Today Osama Bin Laden is still free, but a journalist is in jail and the focus on an international manhunt. One killed innocent people. The other embarrassed our politicians. I can see that Washington values its own pride more than the lives of its citizens.

I watch in horror how politicians have reacted the the challenge of facing the truth behind their actions and words. Wikileaks is not the only people publishing these leaks. They have been reported by many news papers and news outlets. You politicians show your cowardice by attacking the messenger, and attacking an individual because you think you can silence him the easiest. I am sure that Nixon would have loved to arrest Woodward and Berstein as traitors. Clinton would have probably jumped at a chance to silence all the reports of his shady S&L connections. But that isn’t how things work in America, at least it wasn’t how things used to work.

I am a proud American. I stand behind truth, liberty, and the American way. From what Defense Secretary Robert Gates has explained, nothing in these leaks has placed people in harms way. All they have done is expose corruption at great embarrassment.

Are we to be a nation of laws? Or we to be a nation of secrets and deals, civilians sent to die for foreign favors, lies told and lives destroyed?

As my representatives, what will you be doing to restore America’s freedoms and honor?

Reality Deficit

NPR did a great interview with Ayesha Jalal, a professor at Tufts University. She is helping Lahore University create it’s first History program.

What really struck me was how relevant she made the study of history to a nation that seems to be facing so many pressing and current crises. She explained that Pakistan has a “Reality Deficit”.  Without a proper way for people to think about, question, and explore the past; they are left with the simple group-think narratives that are simple and partially true. By only knowing part of the story, they have to fill the gaps of how they got from then to now with conspiracy theories, invisible enemies, and out right lies.

The lack of historical understanding, is a lack of past reality; leaving you with a deficit today. The whole concept is such a great way of stating the obvious disconnect when talking to people…especially about politics and the culture wars.

The simple narrative is that Pakistan was created as a Muslim nation so that Muslims could live free of India. This is the justification of the rising militant movements and their troubled relationship with India. The truth is that their separation is a relic of their common British past and that more Muslims live in India today than in all of Pakistan. This reality of today clashes with the story; so their fill their deficit with conspiracy theories ranging from the mundane to how America is causing the floods.

I think of America and how people think we were founded as a Christian nation with free enterprise  and capitalism are religious ideals. I see how this collides with America really always being a nation of immigrants, never a religious country, and founded by socially conscious elites who never even wanted a democracy. Our own reality deficit is really quite staggering.

Game vs. Play

I wonder what Western extravagance will do to the brains of kids? At all times kids have games. They have pre-packaged sets of rules and goals. All kids have to do is play their games. They play them on computers, on their TVs, on games boards, on their phones, etc.

Poor kids don’t always get games, they just have to play. If you didn’t have a game given to you, you would play house, or restaurant, or just draw. You would have to make up you own rules, determine you own goals. A lot of play isn’t even about achieving a goal, it is about exploring and imagining a bigger world or even a different self.

Sometimes the act of inventing the “Calvinball” is more fun and more important than the game of “Calvinball” itself.

When they grow up, will the children of games be able to live in a world where all of their goals and options not prescribed?