Hadoop Hack-a-thon

Loading Map....

Date/Time
Date(s) - 06/28/2017 - 06/29/2017
All Day

Location
QuintilesIMS

Categories No Categories


About the event

Expenses: No registration fee, meeting venue sponsored by QuintilesIMS, breakfast and lunch are included and sponsored by Cloudera, attendees are responsible for their own travel & lodging

We will not be providing hackathon pre-packaged utilities like containers or synthetic data, please bring your own computers, data, software, knowledge!

The days will include intensive sessions related to OHDSI CDM and OHDSI tools, hands on experts to help debug or solve your Hadoop issues, Atlas on Hadoop, hands on building ETL code, as well as Hadoop 101 training if beginners attend.

Agenda – Day 1

Time Description
8:00 – 8:30am Registration
8:30 – 9:00am Intro from hosts; Goals for the event
9:00 – 1:00pm Break out into different working groups
1:00 – 2:00pm Lunch
2:00 – 5:00pm Break out into different working groups

Working Groups, Day 1

A: Introduction to Hadoop, Spark, SOLR 101 (9 AM – 1 PM)
Presenters: Cloudera
A beginner’s course on Hadoop presented by experts at Cloudera. This session is designed for attendees who may be skilled at OHDSI but beginners to the 20+ tools in the Apache big data ecosystem, or interested in understanding the four or five Apache open source software projects being used by OHDSI members in lieu of Oracle, SQL Server, Postgres or Redshift.

B: OMOP Conversion in Hadoop – ETL Code writing (9 AM – 5 PM)
Presenters: Gregory Klebanov, Kirill Eitvid
Part A – Using the Synpuf data, demonstrate how ETL code can be written in Hadoop. This class will go through how to set up the environment needed and review the ETL code written by Odysseus for the OMOP conversion.
Part B – Breaking out into groups, refactor the ETL code written to improve performance and other issues identified. Outcome of this group is to publish a version of the ETL code that can be used by all member to convert Synpuf data on Hadoop in their environment.

C: OMOP Conversion Best Practice in Hadoop (2 PM – 5 PM)
Moderators: Mui Van Zandt, QuintilesIMS; Odysseus
As a group, talk through what are some of the lessons learned during OMOP conversions using Hadoop. This will cover from the project initiation, architecture set up, ETL mapping, ETL coding, to automation. How can we as a group standardize and share work? As an outcome of this working group, publish at least 10 standard guidelines that the community can use when developing ETL OMOP conversions in Hadoop. These guidelines will not be ‘official’ but will be submitted to the OHDSI community for debate. If time permits, work toward a conversion ‘specification’ or ‘cookbook’ or other tools instantiating those guidelines that might help OHDSI members who need to do a conversion could be created.

Support Professionals (All Day)
Bring your Hadoop/Spark/SOLR OHDSI issues or questions to the F2F. Curious about what tools might e available to help perform conversion/analytics in Hadoop? Whether on OMOP CDM conversion, hardware, cloud, other topics, we will have staff from Cloudera and other organizations that will help you with your Hadoop and other issues. Partnered with each Cloudera staff will be other members of the community who has been doing OMOP CDM conversions for a while just in case you have a question related to OMOP CDM, as Cloudera professionals onsite will have no knowledge of CDM model or SQLRender.

Sign-ups can be for either 30 minutes or 1 hour sessions.

Agenda – Day 2

Time Description
8:00 – 12:00pm Break out into different working groups
12:00 – 1:00pm Lunch
1:00 – 3:30pm Break out into different working groups
3:30 – 5:00pm Review Outcome and Wrap Up

Working Groups, Day 2
A: OHDSI Tools in Hadoop: Architecture, Issues, Work Needed
Moderator: Gregory Klebanov, Tom White (may be remote)
Discuss the issues the community faces when using the ODHSI tools on Hadoop. A perfect example is Atlas. What are the issues preventing ATLAS/WebAPI generated analytical code from working with Impala-enabled OMOP CDM. Take the issues and beginning fixing the issues one by one. If issues cannot be fixed, create a backlog and roadmap on how to fix the remaining issues.

B: OMOP Conversion Best Practice in Hadoop
Moderators: Mui Van Zandt, QuintilesIMS; Odysseus
As a group, talk through what are some of the lessons learned during OMOP conversions using Hadoop. This will span from project initiation, architecture set up, ETL mapping, ETL coding, to automation and export. How can we as a group standardize and share work? As an outcome of this working group, publish at least 10 standard guidelines that the community can use when developing ETL OMOP conversions in Hadoop. These guidelines will not be ‘official’ but will be submitted to the OHDSI community for debate. If time permits, work toward a conversion ‘specification’ or ‘cookbook’ or other tools instantiating those guidelines that might help OHDSI members who need to do a conversion could be created.

Support Professionals (All Day)
Bring your Hadoop/Spark/SOLR OHDSI issues or questions to the F2F. Whether on OMOP CDM conversion, hardware, cloud, other topics, we will have staff from Cloudera and other organizations that will help you with your Hadoop and other issues. Partnered with each Cloudera staff will be other members of the community who has been doing OMOP CDM conversions for a while just in case you have a question related to OMOP CDM, as Cloudera professionals onsite will have no knowledge of CDM model or SQLRender.

Sign-ups can be for either 30 minutes or 1 hour sessions.

Nearby Accommodation & Travel

  • Homewood Suites by Hilton
    • Right next to meeting venue
    • Has free wine and snacks from 4 – 7 PM each day Mondays through Thursdays
  • Hampton Inn Philadelphia/Plymouth Meeting
    • Provides shuttle to meeting venue
  • DoubleTree Suites by Hilton
    • Across the street from the Plymouth Meeting mall which has several restaurants including Redstone, PF Chang and California Kitchen
    • Provides shuttles to meeting venue
  • Courtyard by Marriot Philadelphia
    • Across the street from the Plymouth Meeting mall which has several restaurants including Redstone, PF Chang and California Kitchen
    • Provides shuttles to meeting venue
  • Extended Stay America
  • SpringHill Suites by Marriot Philadelphia

Contact

If you have any questions about the hack-a-thon please contact Gregory Klebanov or Mui Van Zandt

Registration
Registration

Posted in