### Updated PyRAF DBSP Pipeline

When I started doing optical observing in my postdoc, I was unpleasantly surprised at how difficult it was to learn to reduce the data.  Most optical astronomers use a venerable package called IRAF, which may charitably be called “user antagonistic.”  There is a Python wrapper, PyRAF, which mutes some of the annoyances but is no easier to use if you’re not already an IRAF expert.

Using a PyRAF script originally by Branimir Sesar, I extended, generalized, and documented a pipeline (available here) for reducing long-slit spectra from the Double-Beam Spectrograph (DBSP) of the Palomar 200-inch.  It abstracts away many of the IRAF details to enable smooth reduction.  It’s useful both for quick-look classification spectra as well as moderate-precision (few km/s) radial velocity work.  Because it relies heavily on the filename and header conventions of DBSP, it would require extensive revision to use with another instrument.  However, I am advertising it in hope it will be of use to others.

The version (0.2.0) I released today overcomes several annoyances.  It automatically sets the dispersion parameters needed by autoidentify for arbitrary gratings and angles.  I’ve added a modified version of doslit to minimize repetitive prompting of the user (see here for details).  I added quicklook and batch processing scripts for fast, minimally-interactive reductions.  And I expanded the documentation, although it’s probably still not complete enough to help a true novice.

### How is the US astronomy career pipeline changing?

Recently, the American Astronomical Society’s Committee on the Status of Women in Astronomy (CSWA) released a report on the demographics of US astronomers throughout the academic career cycle: graduate students, postdocs, and the various ranks of professors.  The major goal of the report (written by my friend, Prof. Meredith Hughes) was to assess the progress of women through the “pipeline” as a function of time: are women moving into the professor level in proportion to their increasing representation at the graduate student and postdoc levels, or are they “leaking out” of the pipeline?  The full report, summarized in this blog post, addresses this important question.

I was interested in a more basic question: how has the size of the pipeline itself changed over time?  That is, how many more (or fewer) grad students and postdocs are working in US astronomy compared to the number of professors over time?  The report provides proportions of the total number of men and women at each career stage by year (in 1992, 1999, 2003, and 2013), but I was curious about the totals.  Since the survey only covers a limited sample of institutions, it doesn’t represent the totality of the US astronomy job market, but it should provide a useful look.

In [10]:
%matplotlib inline
import pandas
import matplotlib.pyplot as plt
import matplotlib.pylab as pylab

pylab.rcParams['figure.figsize'] = (8, 6)

Below is the data from Figures 2 and 3 of the CWSA report. Because the 2013 report adds 8 universities and 3 research institutes to the 32 institutions surveyed in previous years, I scale the 2013 values down by that fraction. Without the raw data, it’s hard to know whether adding the additional institutions (such as Goddard) bias the career stage proportion relative to previous years. The direct scaling below should provide at least a basic level of year-to-year consistency.

In [7]:
scale_2013 = 32./(32+8+3)
"postdoc":[63,90,137,145],"assistant":[29,45,34,34],
"associate":[18,37,40,44],"full":[23,37,60,71]})
"postdoc":[301,359,473,377],"assistant":[140,212,182,96],
"associate":[162,220,157,187],"full":[421,511,544,426]})
df = df_men + df_women
df.index = df['year']/2
del(df['year'])
df.ix[df.index==2013] *= scale_2013
df
Out[7]:
year
1992 169.000000 180.000000 444.000000 778.000000 364.000000
1999 257.000000 257.000000 548.000000 833.000000 449.000000
2003 216.000000 197.000000 604.000000 818.000000 610.000000
2013 96.744186 171.906977 369.860465 706.976744 388.465116

4 rows × 5 columns

In [8]:
def outside_legend():
# Shink current axis by 20%
ax=plt.gca()
box = ax.get_position()
ax.set_position([box.x0, box.y0, box.width * 0.8, box.height])

# Put a legend to the right of the current axis
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))

First, I plot total numbers of astronomers (men + women) in each career stage by survey year. I’ve lumped the “associate” and “full” professor categories together as “post-tenure,” although this may not be appropriate at all institutions. Note also that for research institutions, “professor” may indicate a staff position.

In [13]:
plt.plot(df.index,df["grad"],"+-",label="Grads")
plt.plot(df.index,df["postdoc"],"+-",label="Postdocs")
plt.plot(df.index,df["assistant"],"+-",label="Pre-Tenure Profs")
plt.plot(df.index,(df["associate"]+df["full"]),"+-",label="Post-Tenure Profs")
plt.ylim(0,1100)
plt.xlabel("Year")
plt.ylabel("Number in survey year")
outside_legend()

To my surprise, the number of astronomers of all career stages appears to have declined relative to a peak in the early 2000s, presumably reflecting flat funding profiles in the US and the lingering effects of the financial crisis.

Comparing raw numbers is somewhat misleading, however, as professors remain in that career stage far longer than postdocs and grad students. We can get a sense of the size of a “cohort” by dividing by a rough average number of years that astronomers remain in a career stage before moving on (or out of the academic pipeline). I have used 6 years for grad students, 3 for postdocs (although taking a second postdoc has become increasingly common in the last two decades), 7 for pre-tenure professors, and 28 for tenured professors.  Dividing by these values gives the rough number of individuals entering (and leaving, in steady state) a given career stage each year.

In [14]:
plt.plot(df.index,df["grad"]/6.,"+-",label="Grads")
plt.plot(df.index,df["postdoc"]/3.,"+-",label="Postdocs")
plt.plot(df.index,df["assistant"]/7.,"+-",label="Pre-Tenure Profs")
plt.plot(df.index,(df["associate"]+df["full"])/28.,"+-",label="Post-Tenure Profs")
plt.xlabel("Year")
plt.ylabel("Individuals per cohort")
outside_legend()

This plot highlights a large excess of postdocs in the early 2000s, presumably due to the influx of funds from to NASA’s Great Observatories (Chandra launched in 1999 and Spitzer in 2003).

Finally, we come to the sticky question: is the pipeline to professorship wider or narrower than it used to be?

Ideally, one would track a defined group through time–surveying the career outcomes of an unbiased sample of PhDs every five years, for example. The CSWA report performs survival analysis using the 1992/2003 and 2003/2013 survey pairs, but the format of the survey can’t ensure that those counted as grads or postdocs at one of the survey periods are among those being counted in the next. (A precocious senior grad student in 2003 could well be tenured by 2013, but would not be counted as “surviving” into the assistant professor stage.)

A more direct means of assessing the width of the pipeline is to compare the relative proportions of grads, postdocs, and assistant professors at each survey interval and assume a steady state. That is, if we keep graduating PhDs, hiring postdocs, and hiring assistant professors/research staff at the rate we are today, what is the oversupply ratio? This is the value most of interest to students already in the pipeline, as it indicates the amount of competition for permanent jobs. (This implicitly assumes the net flux of international astronomers into the US is zero over all career stages.)

In [15]:
plt.plot(df.index,(df["grad"]/df["assistant"]),"+-",label="Grads per new prof")
plt.plot(df.index,(df["postdoc"]/df["assistant"]),"+-",label="Postdocs per new prof")
plt.xlabel("Years")
outside_legend()

If we take these numbers at face value, they suggest that current postdocs face a one in four chance of getting a permanent position in astronomy, while there are more than seven current grad students for every new professor. That’s somewhat worse than implied by older data gleaned from the AAS job register (which has its own biases, particularly in undercounting postdocs). Whether this trend is real, a sampling issue, or a lingering artifact of the financial crisis is not clear.

Current and prospective students must be aware of these numbers and trends in order to be able to make well-informed career choices.   We should also improve graduate training to prepare students for a wide variety of careers, since the majority of astronomy PhDs won’t get permanent jobs in the field.  I’ll outline some possibilities in a future post.

### The Best Books I Read in 2013

Following the 2011 and 2012 editions, here are the most interesting books I read this year:

When God Talks Back: Understanding the American Evangelical Relationship with God
T. M. Luhrmann
How do evangelical Christians in modern America come to believe that God speaks to them directly, as individuals? A tour de force of scholarship, synthesizing history, anthropology, and psychology, with much to offer skeptic and believer alike.

A Grand and Bold Thing
Ann Finkbeiner
A perceptive history of the Sloan Digital Sky Survey, perhaps the most successful astronomical project of all time.  Finkbeiner’s unsentimental rendering of its difficult birth is a clear reminder that science is above all a human endeavor.

The Signal and the Noise: Why So Many Predictions Fail–but Some Don’t
Nate Silver
Why are forecasts better in some fields than in others?  Silver draws examples from a wide range of fields to highlight the importance of rich data, regular feedback, and underlying causal mechanisms and the dangers of out-of-sample predictions and overfitting.

Homeward Bound: Why Women Are Embracing the New Domesticity
Emily Matchar
A balanced examination of why affluent, well-educated young women today are dropping out of the workforce and canning, growing chickens, making meals from scratch, and parenting intensively.  Aptly captures the appeal of the back-to-basics lifestyle (and the role of the Internet in promoting it) as well as potential risks for individual women and communities.

What Money Can’t Buy: The Moral Limits of Markets
Michael Sandel
Argues that the proliferation of market solutions and economic thinking is leading us astray in addressing some moral questions better determined by community deliberation.  In an age of inequality, markets do not neutrally allocate goods, and in some cases applying market logic undermines other values we would like to encourage.  Attempts to persuade gently using many real examples; some are more compelling than others.

Seven Days in the Art World
Sarah Thornton
A deftly composed series of vignettes exploring the sometimes rarefied spheres of contemporary art: auctions, art school “crits,” art fairs and biennales, studio visits, and museum prize exhibitions.  Surprisingly candid interviews with minor and very major players give perspective on the politics but also the pleasures of life in the art world.

### IDL Magics for the IPython Notebook

The IPython Notebook combines code, documentation, and computational results in one package that’s easy to share.  It’s proving a great way to teach, as notebooks are easy for the instructor to write and for students to modify.  Notebooks also provide seamless integration with other programming environments through extensions providing “magic functions“: short code prefixes beginning with % or %% that call other interpreters, like R or Octave.

Since many astronomers who might want to transition to Python already know IDL, I wanted to provide an %idl magic function for IPython so they could easily call existing code.  The difficult interfacing between IDL and Python was already done by Anthony Smith‘s pIDLy package.  (As a bonus, it supports the free GDL interpreter as well.)  I adapted the Octave magic code to provide %idl magic functions with consistent syntax.  You can find the code on github as ipython-idlmagic.

The demonstration notebook below is available on github as well, or you can view it with nbviewer.

## Installation

To begin, we install pIDLy:

pip install pidly


Then we install idlmagic:

In [ ]:
%install_ext https://raw.github.com/ebellm/ipython-idlmagic/master/idlmagic.py

## Usage

When starting a new notebook, we load the magic:

In [1]:
%load_ext idlmagic
IDL not found, using GDL

(I am using GDL rather than IDL on this computer. idlmagic will first look for the idl interpreter on the search path and and fall back to gdl if needed.)

### Line magics

The %idl magic enables one-line execution of IDL commands in the IPython interpreter or notebook:

In [2]:
%idl print, findgen(5)
      0.00000      1.00000      2.00000      3.00000      4.00000

(Note that the %idl line magic fails with TypeError: coercing to Unicode: need string or buffer, dict found in current release versions of IPython (0.13.2 and below) due to a known bug; the github development version of IPython works as expected.)

### Cell magics

Multi-line input can be entered with the %%idl cell magic:

In [3]:
%%idl
x = findgen(5)
y = x^2.
• Maintain your installations manually.  It’s possible–but far more complicated–to maintain your packages manually.  Manual installation isn’t beginner-friendly, though–not all programs are packaged in the same way, and you have to manage dependencies, upgrades, and paths yourself.  If you go this route, pip is currently the installation method of choice, and virtualenv is recommended to create isolated Python environments.  See the installation section of this guide for more tips.
• Use the built-in system Python.  Not recommended!  While both Mac OS X and Linux are likely to have system versions of Python installed, they may change in system upgrades and break your code.  Moreover, the packages you install may cause conflicts with routines your computer depends on.  However, if you just want to play with the language, though, simply typing python in a terminal will give you a way to start.

### Getting a new OS X Finder window in the current Space

Mac OS X provides multiple desktops via the Spaces feature.  I use this feature all the time to keep my mental workspace uncluttered: typically I have one Space with a web browser, one or two more with terminals (iTerm2 is great), and another with documents or presentations.

In Apple’s Spaces model, if you click on the icon of an already-running program in the Dock, it takes you to the Space where that application’s windows are.  For example, if I’m in Space 3 and have iTunes open in Space 1, clicking the iTunes icon will move me to Space 1.

This automatic space-switching behavior is not what I want with Finder.  When I click on the Finder in the Dock I want a new Finder window in the current space.  What happens instead is unpredictable.  If I have no Finder windows open in any space, I get what I want:  a new Finder window where I am now.  However, if I have forgotten Finder windows in other Spaces, OS X will zip me away from what I’m doing to those old Finder windows.  That unexpected switch destroys whatever sense of flow I have as I drag back to my current Space.

Two methods for getting a new Finder window in the current space don’t work well for me personally.  The first is to right click on the Finder icon in Dock and select “New Finder Window.”   The second method is to switch to Finder (either by Cmd-Tab or by clicking on the desktop), then hitting Cmd-N.  Both methods are too cumbersome for me, though they might satisfy you.

Instead, I found an Applescript method to produce a new Finder window.  I modified the code slightly to give focus to the new window and to start in my chosen directory:

on run tell application "Finder" set NewWindow to make new Finder window set target of NewWindow to "Macintosh HD:Users:username:path" activate end tell end run

Open AppleScript Editor (via Spotlight is easist) and paste in the code.  (Replace username with your username and the path as well.)  Save the file as something like new_finder_applescript in your Applications folder.  Set the format to Application and make sure that “Stay open after run handler” is unchecked.

Next, let’s replace the default applescript icon with something better.  I like this CC-licensed Finder icon by Gordon Irving.  Download the .icns version.  Open your Applications folder and Cmd-click on new_finder_applescript and select “Get Info.”  In the top left will be the default “rolled paper” Applescript icon.  Drag and drop the new icon file onto the default icon…

and the new icon should appear.

Finally, drag the new_finder_applescript icon into the Dock.  I like to place it just below the standard Finder (which can’t be removed from the Dock without terminal hacking).

Now, a quick click on your new icon will give you a Finder window in the current Space, no matter what!  It’s possible to bind your script to a hotkey with third-party programs like Quicksilver or FastScripts as well.