Random Forest Analysis of U.S. National Epidemiological Survey on Alcohol and Related Conditions (NESARC) data

In 2001/2002, the National Institute on Alcohol Abuse and Alcoholism (NIAAA) conducted the National Epidemiologic Survey on Alcohol and Related Conditions (NESARC), the largest and most ambitious comorbidity (simultaneous presence of two chronic conditions in a patient) study ever conducted. In addition to an extensive battery of questions addressing present and past alcohol consumption, alcohol use disorders (AUDs), and utilization of alcohol treatment services, NESARC included similar sets of questions related to tobacco and illicit drug use (including nicotine dependence and drug use disorders).

The unprecedented sample size of NESARC (n = 43,093) made it possible to achieve stable estimates of even rare conditions. Moreover, its oversampling of Blacks and Hispanics as well as the inclusion of Hawaii and Alaska in its sampling frame yielded enough minority respondents to make NESARC an ideal vehicle for addressing the critical issue of race and/or ethnic disparities in comorbidity and access to health care services.

NESARC studies the occurrence of more than one psychological disorder or substance use disorder in the same person. In this analysis I have utilized the data collected in the survey for gambling and tried to determine its linkage to other predictors like alcoholic tendencies or drug use or other personality traits that can be detected. General information about the survey can be found here.

The following response variable was predicted: EVER GAMBLED 5+ TIMES IN ANY ONE YEAR (S12Q1).

The following variables (listed below in order of importance) were amongst those included as explanatory variables in Random Forest technique. The accuracy of the random forest was 74% with the maximum accuracy achievable with around 12 trees in the forest. Further increase in number of estimators (trees) did little to increase overall accuracy of the model. On closer examination of the variables found important by the model, we can see that an we can predict about the gambling tendencies of an individual by examining their alcohol consumption patterns, personality and drug use amongst other things.

Variable	Description	Score
S1Q1G	NUMBER OF YEARS LIVED IN UNITED STATES	0.059010
S1Q10A	TOTAL PERSONAL INCOME IN LAST 12 MONTHS	0.057991
AGE	CYEAR (DATE OF INTERVIEW: YEAR) - DOBY (DATE OF BIRTH: YEAR)	0.056678
S1Q1E	ORIGIN OR DESCENT	0.048400
S1Q9B	OCCUPATION: CURRENT OR MOST RECENT JOB	0.046075
BUILDTYP	TYPE OF BUILDING FOR HOUSEHOLD	0.036685
MARITAL	CURRENT MARITAL STATUS	0.029524
SMOKER	TOBACCO USE STATUS	0.028936
NUMPERS	NUMBER OF PERSONS IN HOUSEHOLD	0.027357
S2AQ12B	HOW OFTEN DRANK AFTER MIDNIGHT IN LAST 12 MONTHS	0.025185
S10Q1A52	THE SORT OF PERSON WHO DOESN'T CARE ABOUT WHAT PEOPLE THINK OF YOU	0.024643
S2AQ10	HOW OFTEN DRANK ENOUGH TO FEEL INTOXICATED IN LAST 12 MONTHS	0.024418
S10Q1A16	THE KIND OF PERSON WHO FOCUSES ON DETAILS/ORDER/ORGANIZATION OR LIKES TO MAKE LISTS AND SCHEDULES	0.023065
CHLD0_17	NUMBER OF CHILDREN UNDER AGE 18 IN HOUSEHOLD	0.022352
S10Q1A20	OTHERS THINK YOU HAVE UNREASONABLY HIGH STANDARDS/MORALS/IDEAS ABOUT RIGHT AND WRONG	0.022323
S10Q1A43	ARE THERE VERY FEW PEOPLE YOU'RE REALLY CLOSE TO OUTSIDE OF IMMEDIATE FAMILY	0.022088
S10Q1A25	HAVE OTHERS TOLD YOU THAT YOU ARE STUBBORN OR RIGID	0.021601
S11AQ1A2	EVER STAY OUT LATE AT NIGHT EVEN THOUGH PARENTS TOLD YOU TO STAY HOME	0.020961
NUMPER18	NUMBER OF PERSONS 18 YEARS AND OLDER IN HOUSEHOLD	0.020405
S11AQ1A1	OFTEN CUT CLASS, NOT GO TO CLASS OR GO TO SCHOOL AND LEAVE WITHOUT PERMISSION	0.020345
S10Q1A45	WOULD BE JUST HAPPY WITHOUT HAVING ANY CLOSE RELATIONSHIP	0.020180
NUMREL18	NUMBER OF RELATED PERSONS 18 YEARS AND OLDER IN HOUSEHOLD	0.019861
S2AQ5G	HOW OFTEN DRANK 5+ BEERS IN LAST 12 MONTHS	0.019397
S3AQ3B1	USUAL FREQUENCY WHEN SMOKED CIGARETTES	0.019338
S11BQ1	BLOOD/NATURAL FATHER EVER HAD BEHAVIOR PROBLEMS	0.018869
S10Q1A47	HAVE ALMOST ALWAYS PREFERRED TO DO THINGS ALONE RATHER THAN WITH OTHERS	0.018636
S11AQ1A25	EVER DO SOMETHING YOU COULD HAVE BEEN ARRESTED FOR, REGARDLESS OF WHETHER YOU WERE CAUGHT OR NOT	0.018017
S10Q1A22	HARD TO LET OTHERS HELP IF THEY DON'T AGREE TO DO THINGS EXACTLY THE WAY YOU WANT	0.017820
DGSTATUS	DRUG USE STATUS	0.017671
S1Q1D5	"WHITE" CHECKED IN MULTIRACE CODE	0.017200
S6Q1	HAD PANIC ATTACK, SUDDENLY FELT FRIGHTENED/OVERWHELMED/NERVOUS AS IF IN GREAT DANGER BUT WERE NOT	0.016674
S11AQ1A14	EVER DO THINGS THAT COULD EASILY HAVE HURT YOU OR SOMEONE ELSE, LIKE SPEEDING OR DRIVING AFTER HAVING TOO MUCH TO DRINK	0.016527
S2AQ12F	HOW OFTEN DROVE MOTOR VEHICLE AFTER 3+ DRINKS IN LAST 12 MONTHS	0.016235
S10Q1A58	FLIRT A LOT	0.015546
S3AQ3B2	USUAL FREQUENCY WHEN SMOKED CIGARS	0.014674
S2AQ9	HOW OFTEN DRANK 4+ DRINKS OF ANY ALCOHOL IN LAST 12 MONTHS (WOMEN ONLY)	0.014626
S9Q1A	EVER HAD 6+ MONTH PERIOD FELT TENSE/NERVOUS/WORRIED MOST OF TIME	0.014188
S11AQ1A22	EVER SHOPLIFT	0.014131
S10Q1A3	FIND IT HARD TO BE "OPEN" EVEN WITH PEOPLE YOU ARE CLOSE TO	0.013714
S10Q1A32	OFTEN GET ANGRY OR LASH OUT WHEN SOMEONE CRITICIZES OR INSULTS YOU	0.013670
S10Q1A46	TAKE LITTLE PLEASURE IN BEING WITH OTHERS	0.012579
S11AQ1A15	EVER GET MORE THAN 3 TICKETS FOR RECKLESS/CARELESS DRIVING, SPEEDING, OR CAUSING AN ACCIDENT	0.012403

The code and output is included below

Confusion Matrix: 

[[11983   892]

 [ 3540   823]]

Accuracy Score is:  0.742893607147

The quick and dirty on getting a Hadoop cluster up and running

The last time I tested out a hadoop cluster, it was about four years ago. At the time, the setup was manual and I used two machines to set up a two node cluster. I was able to run map reduce jobs and test the system out. Fast forward to 2017 - there are a lot more animals in the Hadoop circus and cloudera has made it very convenient by having various options to test - a quickstart VM, docker images etc. Still getting the system up and running and executing the first sqoop job took some effort. I have documented what I did including the tweaks so anyone running into the same can get help. Here are the steps:

1. Get a system ready - at least 16GB or RAM. I had a Linux Mint box on hand that I used. Mint is generally similar to Ubuntu but you do have to watch out for version specific instructions. 
2. The next step is to install Docker. For Linux Mint 18, the steps here from Simon Hardy came really handy.
3. Cloudera provides a Docker Quickstart container - do not use that. There is a lot of documentation and links on that and its quite easy to go down that path. A better option is to use the Cloudera Clusterdock. Clusterdock is a multi-node cluster deployment on the same Docker host (by default it does a two node). The clusterdock documentation is very useful but there are a few catches that I will note here. There are few other links on clusterdock here:
1. The cloudera online tutorial is based on the quickstart docker container or the quickstart VM. There are several dependencies including a mysql database, flume files that are used in the demo etc. I would suggest that you keep that container also around for a bit and copy the data as needed to the clusterdock nodes. The clusterdock setup is much more stable in the cloudera manager and the slight inconvenience (occasional hardware freezing) may be worth it. After launching the quickstart container, you may simply tar the /opt/examples folder and the mysql retail_db database and transfer that to the host machine using the docker cp command and then kill the quickstart container.
2. The clusterdock.sh script on the cloudera website lacks 'sudo' in a couple of places. Be aware of that. Its easy to spot that in case it causes a problem. For example the ssh function has this problem.
3. I wanted to run the sqoop command and for that I needed a myql database to connect to. There was one I installed on the host machine. It was almost impossible or looked very time consuming to connect the clusterdock containers to the host mysql. You run into a docker networking issue. The easy way out is to install a mysql docker container and put it on the same user defined network that the clusterdock nodes use. Note that you have to force the IP and the network on the mysql container to do that and also map the mysql port 3306 so its open for access. Else you will waste a lot of time!
4. Next step is to ssh on the master node and run a sqoop job. At this time you will run into a lot of permissions issues if you are not careful with where you are storing the target imported files. sqoop will generally report exceptions stack trace with these permissions errors. Best is to google and fix the any paths you give to the sqoop command.
5. You also need to copy the mysql jar file in the sqoop lib folders. Easiest way is to get it on the host box and then use the docker cp commands to move it to the desired location.
6. That should do it - get you past the sqoop step and then you can run a query in Hue.

Analyzing Gapminder Data

Founded in Stockholm by Ola Rosling, Anna Rosling Rönnlund and Hans Rosling, GapMinder is a non-profit venture promoting sustainable global development and achievement of the United Nations Millennium Development Goals. It seeks to increase the use and understanding of statistics about social, economic, and environmental development at local, national, and global levels. Since its conception in 2005, Gapminder has grown to include over 200 indicators, including gross domestic product, total employment rate, and estimated HIV prevalence. Gapminder contains data for all 192 UN members, aggregating data for Serbia and Montenegro. Additionally, it includes data for 24 other areas, generating a total of 215 areas.

GapMinder collects data from a handful of sources, including the Institute for Health Metrics and Evaulation, US Census Bureau’s International Database, United Nations Statistics Division, and the World Bank.

Decision tree analysis was performed (using python) to test nonlinear relationships among a series of explanatory variables (0-13 below) and a binary, categorical response variable (above average suicides per 100,000 in 2005). Note that python does not support pruning. The code is included below the analysis.

Index	Variable	Description
X[0]	income per person	2010 Gross Domestic Product per capita in constant 2000 US$
X[1]	alcohol consumption	2008 alcohol consumption per adult (age 15+),
X[2]	armed forces rate	Armed forces personnel (% of total labor force)
X[3]	breast cancer per 100th	2002 breast cancer new cases per 100,000 female
X[4]	co2 emissions	2006 cumulative CO2 emission (metric tons)
X[5]	female employment rate	2007 female employees age 15+ (% of population)
X[6]	HIV rate	2009 estimated HIV Prevalence % - (Ages 15-49)
X[7]	internet use rate	2010 Internet users (per 100 people)
X[8]	oil per person	2010 oil Consumption per capita (tonnes per year and person)
X[9]	polity score	2009 Democracy score (Polity)
X[10]	residential electricity consumption per person	2008 residential electricity consumption, per person (kWh)
X[11]	employment rate	2007 total employees age 15+ (% of population)
X[12]	urban rate	2008 urban population (% of total)
response	suicide per 100th	Is the 2005 Suicide, age adjusted, per 100 000above average? (True or False)

Alcohol consumption was the first variable to separate the sample into two subgroups. The threshold was identified to be 16.19 litres and countries where it was higher than that had a likelihood of above average suicide rate. Next one was the amount of electricity consumed in residences. If the consumption was below 230 kwh, there was a likelihood of higher suicide rates. This was followed by armed forces rates - if less than 41% of the population was part of armed forces, the likelihood was higher. Alcohol consumption and employment rate comes next - higher amounts of alcohol consumption (higher than 14%) and a low employment rate (lower than 49%) had the likelihood of an above average suicide rate. Countries with low alcohol consumption (14%) and low female employment rates (less than 56%) had lower likelihood of a below average suicide rate. However, if this is combined with low internet use (lower than 85%) this led to likelihood of above average suicide rates.

The total model classified 64% of the sample correctly.

Dumping Tomcat packets using tshark

Tshark (command line version of wireshark) is a wonderful tool for dumping packets and recently I used it on my Mac since I couldn't easily get Tomcat to log the HTTP packets coming in on port 8080. Having used it in the past for lots of other reasons, I felt compelled to find a generic solution to this problem where you have to rely on application level logging to determine why something works or doesn't.

Here is the command I used (lo0 is the loopback interface since I was running the client and server on my PC)

tshark -f "tcp port 8080" -i lo0 -V. Here is a very good page on tshark that I am sure I will come back to again and again to get more juice out of this tool.

Getting Postman to test Restlet

Postman is a wonderful Chrome application for testing REST method calls. However it was a bit hard to get it to work for testing Rest calls with my Restlet server. Here are the gotchas I faced and havent yet figured them out completely. I love to record them so I can come back much later and they are still here ;)

1. Postman does not add the Content-Type header by itself. If you select an HTTP method which allows a body (e.g POST), it allows you to create the body and select the format (JSON/XML etc) but you must remember to add the Content-Type header.

2. If your application requires authentication, you can add that too. Postman supports basic, digest as well as OAuth (which I would love to test out next).

3. The biggest problem I ran into was when I sent XML or JSON body and the Restlet server replied back with 415 - Unsupported Media Type. The request does not even hit my application side! If you write a client application and choose the Media Type of MediaType.APPLICATION_XML and the server side method is annotated by @Post("txt:xml") it works. However, when you set the Content-Type header in Postman to application/xml it does not work. To debug this further, I installed wireshark and dumped the packet contents. I was surprised to find that the client built using Restlet framework was sending a Content-Type header as text/plain. This had to be some issue on my end. Interestingly, if I made the corresponding changes in Postman, the request coming out from that application also started to work. These are the two headers I inserted. Note that a Content-Type of text/plain still does not work. You must indicate the charset as well to make it work.

Content-Type: text/plain; charset=UTF-8

Accept: application/xml