Here is an overview of my solution.
- [Instructor] For this solution I felt that querying the website to display results to the user would be fundamentally fragile and slow. Baring this in mind, I considered two possible solutions, both using the idea of downloading the weather data in advance and querying against that. One solution would be to download the text files and have our program load those directly to calculate the means and medians. I decided against this because I thought the performance would be poor especially for queries spanning multiple years. The program would need to load and process many megabytes of text files.
Instead I decided to download and store the data in SQLite to create a local database that is both faster to query and would demand less memory overhead and less total storage. Instead, all calculations are performed against this database. The database comes with the program and contains the first several years of data. We'll also create a script to pull more recent information as needed. This means that the mean and median calculations never have to wait on time-consuming web requests and can happen directly inside of the database.
For this solution Clojure and it's ecosystem have a few helpful features. Clojure's features for lazy evaluation make it easy to process data as it's coming in rather than waiting for the full body of the response. For the SQLite portion I knew that I could leverage JDBC to do aggregate queries, and in certain operations in bulk, for performance gains that are easy to program without a lot of manual optimizations. This solution relies on a handful of third-party open-source libraries to operate. For the database portion the Clojure java.jdbc library is used with a SQLite driver.
Web requests are made using clj-http and datetime operations are performed using clj time. As with all the solutions in this course, the dependencies are listed in the Projects field J file in the chapter folder. These will automatically be downloaded and installed whenever we need to use them inside of Lienkin. The solution consists of four different files or name spaces. The Request name space is responsible for fetching weather data from the Navy's website. The DB name space is responsible for defining the procedures that update or query data in the local SQLite database.
These two name spaces are independent of each other so the Backend name space is used to tie them together. This name space uses the functionality provided by each of these to define the core logic of our program and exposes them in a clean programatic interface to the Core name space. Lastly, this Core name space defines a command line interface for the program. In the following videos for this chapter we will write each of these pieces in turn and demonstrate how each of them contribute to the overall solution. Let's get programming in Clojure.