So you want to do a PhD?

Here is a brief extract of comments I made in an interview in Singapore this year. 

The first piece of advice I give to PhD students is to not be in a big rush to get the data and do something with it. You have to have a really good question first. Otherwise, no matter how good your data is, your work is not going to be interesting and you will not publish a good paper. 

The second thing is, you should always theoretically analyse the question completely first because that exercise will take you to the right empirical specification for the data work that you want to do. There should be a good theory because once you have a theory, then you can derive a setup that tells you that if this assumption holds, then you should see this in the data, etc. “No theory, no paper.” 

Another piece of advice is not to be in a big rush to find a topic because you are going to be stuck with it for the duration of the PhD and possibly a few more years. You do not want sub-optimally close on something that you will not enjoy. 

Additional advice is to start writing right away, do not wait; the process of writing helps you structure your thoughts better, structure your theory better, structure the empirical work better. Of course you should always be ready to throw things away. 

You should write early rather than late and keep writing. That is part of the life of a researcher; writing a lot is a good thing and throwing away a lot is also a good thing. And you know, you keep learning through that process, and that is important.

You should not expect to learn everything you need to know in a PhD programme; that is never going to happen. There were countless things I learnt after my PhD because I needed to solve a problem, I needed a technique that I did not know and I had to learn it for myself; you should be learning how to learn instead of thinking that everything is going to happen in the PhD programme. That is the purpose of the programme.

Research Environments

What makes a research environment in a business school thrive? Not the same stuff that makes firms profitable or breeds winning sports teams. It would be too easy.

The good and the bad: Consider the following scenarios I see played out at campuses across the country: Just recently someone asked me – “Why do you come in to work when you are tenured?” He looked at me as if I was irrational and that I should be away somewhere lying on a beach or making heaps of money by starting a company. And again, another junior professor inquires of a tenured one – “Why are you in today, when you are not teaching?” I have noticed at many universities that junior faculty do not go to seminars as much as senior ones. Is this a permanent cultural shift in academia? How come junior faculty are not being shepherded by seniors as they used to be in the old days? Why are there so many empty offices, and so little discussion in the hallways? Is this a reflection of the new generation of young faculty, or is it a reflection of norms gleaned from the senior ones?

There are warning signs, but also signs of encouragement and hope. There is more collaborative work than before, a greater number of seminars, and higher promotion standards, as the old excuses for the inability to publish are being shelved, as a few brave young faculty show us all how it can be done, even as journal space becomes scarcer and aggressively fought over.

A Categorization of Faculty: We may think of research faculty along two dimensions: (a) Activity: producing high quality knowledge, and (b) Presence: engaged with research activities. Just being research active is not enough, being present to facilitate the research environment is important. Viewed along these two dimensions gives us four types of faculty: {Active, Not Active} x {Present, Not Present}. I suppose it is easy to categorize any faculty into these four buckets. What type are you? Which bucket do you fall in?

The {Not Active, Present} category is more common than we might imagine. This is part of a natural cycle where Active faculty burn out and yet stay in the game by being Present. Indeed their experience is still valuable as they maintain standards, carry valuable institutional memory and perspectives, mentor junior faculty, and can judge work to ensure that promotion standards are not lowered. This is also often a temporary phase where folks take a pause to recharge batteries and then embark anew, often on something fresh.

The {Active, Not Present} group presents an interesting challenge. How does the organization engage them so that they are more than the sum of their own activity? Being present means fostering spillover benefits. But make no mistake, being Present is a double-edged sword, it is costly to Active faculty but also comes with benefits. Too much of it can be counterproductive. But, none of it is more surely a dead loss to the research organization and the researcher as well.

Outcomes: There are several benefits/costs of tenured faculty being Present, depending on which way you look at it. Here are some:

1. Mentoring junior faculty.
   
2. Working with junior faculty and enhancing tenured faculty Activity.
   
3. Sharing the burden of ad-hoc demands on the faculty that otherwise have to be borne by the Present set.
   
4. Quicker and higher quality group decisions in a face-to-face manner.
   
5. Keeping tenured faculty expertise “on-line” for access by others.
   
6. Better utilization of office space, a scarce resource in academia today.
   
7. Demonstration effect/Building culture: setting a good example and high expectations for the next generation.
   
8. More attendance at seminars to improve quality of discussion, and show a strong presence to outsiders so as to leave a good impression of the school’s scholarship quality and environment.

There are several flavors of Present. Some faculty are introverted but work intensely with a few junior faculty. Others help a broad swath of faculty or resolutely maintain standards. Some don’t come to work a lot, but come to all seminars and engage with juniors. And so on. We academics are all unique and we must each choose our own brand of Present.

Assume a highly Active and Present senior faculty. What features of the junior faculty would be evidence of the benefits of such a research environment?

1. Higher quality and quantity of scholarship.
2. Junior faculty who actively talk about their research with other faculty. This enhances the general awareness of what everyone is working on, and brings benefits of several faculty offering suggestions and pointing out related research. A self-exciting and perpetuating research atmosphere.
3. Junior faculty who give their papers to others to read, actively asking for comments.
4. Junior faculty that actively seek out engagement with practitioners, and continually ask themselves how they might influence practice.
5. Junior faculty that actively take steps to enhance the research environment by organizing seminars, inviting special speakers, setting up labs, engaging in popularizing scholarly work that transcends mere publishing.
6. Confident junior faculty who actively demand more resources for their research, and also generate their own means and funding.
7. Junior faculty who work with faculty in other departments and disciplines.
8. A greater number of submissions to top journals.

Making progress: How does a school take steps towards this idealized environment? First, measurement. Take stock by classifying existing faculty into the four bucket model. This is easy for department chairs to do. Place each faculty person on the 2×2 {Active, Not Active} x {Present, Not Present} grid. What is a reasonable number in each bucket?

Second, self-realization. Discuss the grid with the faculty and ask them to think about where they are and where they might want to be.

Third, incentivize movements into the {Active, Present} bucket. Use a coarse system of raises. Faculty in the {Not Active, Not Present} bucket get no raise, and in fact should be moved into shared offices. Faculty in the {Active, Present} bucket should get three times the raise of those on the {Active, Not Present} and {Not Active, Present} buckets, recognizing that being Active and Present is worth more than just the sum of the two dimensions.

Fourth, raise salaries to market so that departments can compete in the marketplace, and raise funding to support research so that junior faculty are able to compete with others from other schools.

Lesson: There is a tide in the affairs of research schools, when taken at the flood, leads on to fortune.

The Less You Know

The more you learn, the less you know.

What? How can this be? Some sort of weird paradox? No — the more you learn, the vastness of what you do not know becomes more apparent, and hence, you become increasingly aware of how little you know. The ignorant are unusually blessed for they know not what they do not know.

I think it is true to say that knowing what you don’t know is even more important than what you know. So many mishaps have occurred because we assumed we knew, when we deep down we knew we didn’t. You should not make a virtue of trusting in your ignorance. Instead, admit that you don’t know, because it is the first step in knowing deeply.

Half-knowing is just as bad. But not knowing and assuming that it’s okay is worse. I’d feel much safer with a doctor who would tell me he did not know what was wrong with me than with one who said he did when he didn’t. Or a lawyer for that matter who gave me false hope or advice. Because in the end, the truth is most important, and someone who leads you on as if he/she knows pushes the truth further away, and in some cases, such as medicine, it can be quite dangerous.

Admitting one’s ignorance to one’s self is very hard, and often our egos won’t allow it. But letting in the admission is winning the battle. It marks the beginning of the learning process. Sometimes you have to learn it the hard way. I remember the first few months of my graduate education at Berkeley when I was studying computer science. I was the only one in class with no background in engineering or computers, so was pretty lost in those early weeks. I was floundering and not sure what to do as jargon and terminology, as well as math, kept whizzing by. Having had plenty of experience with continuous math, I found discrete math hard, but failed to admit it, so I just assumed I should know, and that I did know. Yet, I didn’t and nothing would change the facts. I was stuck and not learning.

In the end my frustration got me to admit to myself that I did not know a lot. I began to carry a little notebook around to jot down every word and concept that I had no clue about and that I needed to know. I would return to my cubicle after class and then bug whoever was around to tell me about the new unknowns in my notebook. This little book was my confession of how little I knew. It was also my salvation. I went home every day enlightened by answers to the ever-growing list in the little notebook. It became the symbol of my salvation, my new found knowledge. I began to feel good about how little I knew, because it was gratification wrapped up in humility. The less I realized I knew, the happier I became.

Make a list of stuff you don’t know (see http://swanson.github.com/blog/2011/12/04/whats-on-your-learning-list.html )– it is your gateway to knowledge. Of course, it is impossible to make an exhaustive list, it would take forever. So make a list of things you do not know, and are interested in. Keep it short. Even if there is just one thing on that list but you spend time to get to know it well, it will be exhilarating.

Framing

I love spending time looking at art in museums or galleries. It’s like a big buffet and you can consume more of what you like, and sample some of the other offerings. And just like a good meal, I am satiated, tired, and happy at the end of the indulgence. Mentally and emotionally, that is.

I was with a friend recently at the DeYoung museum who commented on the excessive ornateness of the frames on the art, which distracted and detracted from the beauty of the painting itself. And it struck me, literally and figuratively, how much framing matters.

We exhort ourselves to never judge people by their looks, or a book by it’s cover, but at the end of the day, we succumb to framing. Advertisers have been exploiting our shallow judgment heuristics for years.

So, when given a choice to frame something like a great work of art in good light, why do we choose bad framing? One can understand the opposite, where framing can be used to improve a poor impression, but adverse framing is harder to reconcile.

Really good art should have no frame, just like a truly beautiful woman needs no make up. And closer to my own field, a truly original idea does not need to be dressed up in an excessive number of mathematical equations. And yet, so many beautiful women overdo their face packs, and research papers are written in trappings that obfuscate and confuse, rather than make us more knowledgeable. Why?

At some level we are all insecure, because we do not really know that we are already worth a lot just as we are. So we err by overdoing our framing. We end up not enhancing but cluttering. Like a house with too much furniture or art on the walls that feels less like home and hard to live in, we become uncomfortable in our trappings, and deny the pureness of our own skin and being. This only makes us more insecure and perpetuates the excess framing cycle.

Or, we play the frame game. Signaling becomes the goal of framing. Form over substance. It is why we need to wear an expensive business suit to meet a client, to show we are serious and the client is important. That in itself is not a bad thing, but the client really begins to believe we are more qualified than someone who could not afford the same expensive suit. The converse is worse. When we do not wear the expensive suit even when we are better qualified, that we are downgraded, to everyone’s detriment. Framing to signal is deep-rooted in nature. Birds with better plumes attract better mates. It’s a time-tested outcome of Darwinian evolution. It’s when we try to do more than nature prescribes that we make a mess of things. And when we do it collectively, kowtowing to the exaggerated norms of society, we make things even worse!

We are a strange collection of paradoxes. When we are supposed to be more creative, as in the art realm, we end up conforming more. Art is heavily framed because that’s the way it’s done. No ifs or buts. Casual Fridays exist, but not casual Wednesdays, which I think would be much nicer! But the former has a frame of precursing the weekend.
So it’s acceptable.

Yes, I know I am exaggerating a bit. Frames can be utilitarian. They protect art. Our clothes are frames, to protect our sensibilities. This is what nature intended. But we are cursed to be fooled by frames, and also to indulge in bad framing. Maybe that’s also what nature intended?!

Web-enabling R functions with CGI on a Mac OS X desktop

I write many R functions for my own use and for use in class. I have been making these functions available from a web page for some time, and finally decided to just post a simple example to make it easy for others to do the same. This is just an example based on the “Rcgi” package from David Firth, and for full details of using R with CGI, see http://www.omegahat.org/CGIwithR/. Download the document on using R with CGI. It’s titled “CGIwithR: Facilities for Processing Web Forms with R”.

Of course, if you don’t have R at all, then download R and install it from http://www.r-project.org/. Then use the R package manager to install the Rcgi package.

You need two program files to get everything working.
(a) The html file that is the web form for input data.
(b) The R file, with special tags for use with the CGIwithR package.

Our example will be simple, i.e., a calculator to work out the monthly payment on a standard fixed rate mortgage. The three inputs are the loan principal, annual loan rate, and the number of remaining months to maturity.

But first, let’s create the html file for the web page that will take these three input values. We call it “mortgage_calc.html”. The code is all standard, for those familiar with html, and even if you are not used to html, the code is self-explanatory.

<html>
<head>
<title>Monthly Mortgage Payment Calculator</title>
</head>

<FORM action="/cgi-bin/R.cgi/mortgage_calc.R" method="POST">
<body>
Loan Principal: <INPUT name="L" value="" size=5><p>
Annual Loan Rate: <INPUT name="rL" value="" size=5><p>
Remaining months: <INPUT name="N" value="" size=5><p>

<P><INPUT type="submit" size=3>

</body>
</html>

Notice that line 06 will be the one referencing the R program that does the calculation. The three inputs are accepted in lines 08–10. Line 12 sends the inputs to the R program.

Next, we look at the R program, suitably modified to include html tags. We name it “mortgage_calc.R”.

#! /usr/bin/R

tag(HTML)
	tag(HEAD)
		tag(TITLE)
			cat("Mortgage Monthly Payment Calculator")
		untag(TITLE)
	untag(HEAD)

tag(h3)
	cat("Mortgage Monthly Payment Calculator")
untag(h3)

lf(2)
tag(BODY)

tag(p)
	tag(b)
		cat("Inputs:")
	untag(b)
	
	tag(p)
	L = as.numeric(scanText(formData$L))
	cat("Loan Principal: ")
	cat(L)
	
	tag(p)
	rL = as.numeric(scanText(formData$rL))
	cat("Annual Loan Rate: ")
	cat(rL)
	
	tag(p)
	N = as.numeric(scanText(formData$N))
	cat("Remaining months: ")
	cat(N)
untag(p)

lf(2)
tag(p)
	cat("Monthly Loan Payment: ")
untag(p)

r = rL/12
mp = r*L/(1-(1+r)^(-N))
cat(mp)

untag(BODY)
untag(HTML)

We can see that all html calls in the R program are made using the “tag()” construct. Lines 22–35 take in the three inputs from the html form. Lines 43–44 do the calculations and line 45 prints the result. The “cat()” function prints its arguments to the web browser page.

Okay, we have seen how the two programs (html, R) are written and these templates may be used with changes as needed. We also need to pay attention to setting up the R environment to make sure that the function is served up by the system. The following steps are needed:

1. Make sure that your Mac is allowing connections to its web server. Go to System Preferences and choose Sharing. In this window enable Web Sharing by ticking the box next to it.
2. Place the html file “mortgage_calc.html” in the directory that serves up web pages. On a Mac, there is already a web directory for this called “Sites”. It’s a good idea to open a separate subdirectory called (say) “Rcgi” below this one for the R related programs and put the html file there.
3. The R program “mortgage_calc.R” must go in the directory that has been assigned for CGI executables. On a Mac, the default for this directory is “/Library/WebServer/CGI-Executables” and is usually referenced by the alias “cgi-bin” (stands for cgi binaries). Drop the R program into this directory.
4. Two more important files are created when you install the “Rcgi” package. The CGIwithR installation creates two files:
   (a) A hidden file called “.Rprofile”
   (b) A file called R.cgi

   Place both these files in the directory: /Library/WebServer/CGI-Executables

   If you cannot find the .Rprofile file then create it directly by opening a text editor and adding two lines to the file:

   #! /usr/bin/R
   library(CGIwithR,warn.conflicts=FALSE)

   Now, open the R.cgi file and make sure that the line pointing to the R executable in the file is showing

   R_DEFAULT=/usr/bin/R

   The file may actually have it as “#! /usr/local/bin/R” which is for Linux platforms, but the usual Mac install has the executable in “#! /usr/bin/R” so make sure this is done.

   Make both files executable as follows:

   chmod a+rx .Rprofile
   chmod a+rx R.cgi

5. Finally, make the ~/Sites/Rcgi/ directory write accessible:

   chmod a+wx ~/Sites/Rcgi

Just being patient and following all the steps makes sure it all works well. Having done it once, it’s easy to repeat and create several functions. You can try this example out on my web server at the following link.

The inputs are as follows:

• Loan principal (enter a dollar amount)
• Annual loan rate (enter it in decimals, e.g., six percent is entered as 0.06)
• Remaining maturity in months (enter 300 if the remaining maturity is 25 years)

Asynchronicity

The world is becoming ever more asynchronous. We do many things together, but this has become increasingly likely to be done at arms length. Face to face meetings have become less likely. Even worse, instead of talking on the phone, an email often suffices. Everything has become asynchronous. Truth be told, we probably like it this way!

There are examples everywhere. Instead of playing cards by sitting around a table, we now play internet poker. The same is true of chess, which is not only online but asynchronous, yours truly being a shining example of succumbing to this phenomenon. Instead of the phone, we send emails. Even TV watching, which used to be a joint family past time is now relegated to individual laptops in separate rooms. Instead of the entire nation watching a TV program at the same time, DVR technology has ensured that we all watch it on our own time. Even sports is watched with time delay in so many locations.

But making it convenient to consume entertainment has made it inconvenient for us to spend time together. We are all running to complete the ingestion of content, leaving little time for blank moments when we might spontaneously interact with each other. Is there no way out of this mess?

Here are some ways to fight this, for in this case the trend is not your friend.

1. Consume less media. Most media consumption is now asynchronous and done independently of others. We do not watch the news together, not even sports. So just consume less of it. That goes directly to curtailing asynchronous consumption of media. Watch as much live as possible, with someone else. News and sports are ideally suited to this approach.
2. Stop recording. It isn’t that hard. Just get rid of the DVR. This will also help in reducing the vast amounts of time spent on TV. It will also help you do just one thing at a time. I began taking my ipod along on a walk to listen to podcasts, and as a result stopped looking around and enjoying nature. I just missed out on the peaceful quiet on my night walks, and I did not realize how much I had enjoyed it till I stopped taking the ipod with me.
3. Switch of all cell phones, computers, and singular distractions after some specific time each evening. This really works. My reading went up three-fold once I took this step. And my sleep was much better. There is plenty of evidence that imperceptibly flickering screens can mess up sleep for several hours. After switching off screens, I was not sleeping much more, but my sleep was of much better quality.
4. Produce something every day in place of consumption. Instead of only reading, write something, and I do not mean emails. Responding to emails is not “producing” anything, and it does not bring deep satisfaction. But writing, even something trivial like a blog post, feels really good.
5. Play team sports, and i don’t mean MPOG (multi player online games). Getting exercise this way is much better than the isolating act of going to the gym and pounding a treadmill alone. There is so much more stimulation getting exercise in groups. Even just hiking can be so much more than just an exercise in exercise. Do things with your hands where community is required, for example gardening clubs.
6. Join a few meet up groups. Meetups are cool, new phenomena where interest groups organize get-togethers using web technology. The meetings are in person and synchronous.

Synchronicity is about community, and community is very important. However, we seem to be slipping into a world of asynchronicity. The good news is that this problem is beatable, one person at a time. As everyone, one by one, starts engaging in synchronous activity, we unwind asychronicity rapidly, because when people do things together, a network builds rapidly, and network effects rebuild synchronicity.

Wracker

I love writing and I also love programming, but I am not very good at either. I’d say my skill level is fair. But who says one must be good at the things you love? 

As one grows older, one experiences a growing unease, a loss of anchoring that makes for deep dissatisfaction. Having passed the usual thresholds of ambition and need, happiness comes not from being good at things, or better than others, but from doing what you love. 

This simple realization came to me as I was taking a walk down Broadway in New York. My alter ego tapped me on the shoulder and asked why I did not indulge in writing and hacking? And the truth is, I don’t know. The truth is, I love writing and hacking, but have been distracted with stuff like Internet, TV, work. So maybe I need to be a “wracker”  — someone who writes and hacks!

It’s an act of pure creation, unlike TV, Internet, and some sorts of work that are, in essence, mere consumption. So it seems, real satisfaction in life comes from producing cool new things, not just from consuming. But no economist knows this secret! That a large part of utility comes from the opposite of what goes into the mere consumption of things.

It’s easy to be a wracker nowadays since the internet makes both pursuits available easily, in the form of a blog. It’s like being able to be a short-order cook. So I plan to write my blog more often, but will also open up a new channel with annotated program code that I develop and post to my blog for others to use.

It’s a subtle thing, and fine balance, but when production displaces pure consumption, that’s when we have true satisfaction.