Spark architecture for genomics

- [Instructor] In this next scenario,…we're going to look at genomic variant pipelining…that includes Hadoop and Spark.…In earlier movies in this course,…we talked about augmenting the Hadoop library,…such as Spark,…with additional open source or commercial libraries…and actually showed and talked a little bit…about ADAM for genomic processing.…You may remember that the ADAM set of libraries…which wrap around Spark…include domain specific implementations of items…such as schemas for the incoming files…which are of a specific format.…

You can see SAM, BAM, or VCF.…These files will be coming in…from genomic sequencing machines…such as those made by Illumina.…These is a simplified pipeline.…You see the source files coming directly into Amazon S3.…This is an Amazon implementation…and then the focus here is showing…that the ADAM libraries are running on top of…an Amazon EMR cluster which is running Spark.…In the real world, you would probably have a number…of pre-processing steps before the files were processed…by the Spark cluster.…