Easy-to-follow video tutorials help you learn software, creative, and business skills.Become a member

Processing the full-year records

From: Code Clinic: PHP

Video: Processing the full-year records

We've downloaded copies of the full year data for 2001 to 2010. If you click inside line one between any of the Lets create a script to start extracting the data so we need a new PHP file.

Processing the full-year records

We've downloaded copies of the full year data for 2001 to 2010. This is 2001.txt. Line one is a series of headings and the data is stored in columns below. But if we scroll across to the right, we can see that the headings and the columns aren't aligned. So we'll need to inspect the data in here a little bit more closely before we can extract the data that we need. If you click inside line one between any of the headings and press the right arrow key you move one space at a time, but if you click between two columns of data and press the right arrow key, you jump right across.

So that means we're dealing with tab separated data. Lets create a script to start extracting the data so we need a new PHP file. And we call process_years.php. And I'll see that it's ended up in my RAW data folder, it needs to be in the top level folder, so I'll move it to problem one and then we can start work on the script. First thing we need is variable for the year, we're going to start with 2001 and then we need to open the file.

We use the Fopen function for that that returns a reference to the file. First target to F open is the file location it's in the raw data folder and the file name is the year followed by the .txt. And the second argument to Fopen is the mode, we want to read it. So the second argument is String with r inside it. We can get the first line using the Fgets function which gets one line at a time.

Takes the reference to the file, and then the next line we're dealing with tab separate values. So we can use the Fgets CFV function which deals with character separated values including tabs. Install the result as line. First argument is the reference to the file. The second argument is the maximum number of characters in the line, we'll say 500. The third argument is the delimiter or what is separating the values, that's tab.

So we need, double quotes backslash t. Then we can close the file. And Fgets CSV returns an array of values so we can inspect that array using the print r function. So what we're looking at is line. So if we save that and then view it in a browser. There is the array that is being returned. The important thing to notice is that the first element is both the date and the time as a single element. And then all the other elements are separate.

And we're going to need to separate the date from the time. So we need to count the number of characters there it's four plus two plus two. And then the underscore so we've got a total length of ten characters. So let's go back to the editing program. And if we look in 2001.txt they can time that's the first element zero, wind speed is one of the values we want that's at index one. Air temperature is also a value that you want.

That's at index four and if we scroll across, barometric pressure is at index six. So, we need to create variables for those indexes. So, we'll have wind, we'll just call it 'W' That was index one. Air was index four. And barometric pressure was index six. The code online eight gets only one line at the moment.

So we need to put that into a while loop. So while and then we need to have two parenthesis there. And we're going to put another pair of parenthesis at the end there. And then not equals false. So there is our condition for the while loop. So what will happen here is that we will get one line at a time. But when we get to the end of the data, Fget CSV will return false, and so the while loop will come to an end.

First thing we need to do inside the while loop is to get the date, which we can use using the substring function. So, date equals and then substring, and what we're looking at is the first element in line. So that's line zero, and then the start is right from the beginning, we count from zero, so it's zero. We counted ten characters in the date, so the length is ten.

And we can now use this to create a multi dimensional array. So lets initialize that array outside of the while loop. We'll call it data set and we will initialize it as an empty array. And then we can start building the multi dimensional array in the while loop. So our data set will have the date, then the sub-array will be air temp, Underscore there and then a pair of empty brackets at the end.

That will be the next element that will go in there. And we assign to that, the value of line with the value of air. The index of air is stored as A. And I've discovered that some figures have a leading space. So we'll pass each of the figures to ltrim to remove the leading spaces. So Ltrim, and then it is line and then its index is stored as A. So we need to copy that line twice, and then we'll change that from air temp to bar press and the index is B.

And then this one will be wind speed and that index will be W. So, we can now test this script by changing the value past the print r to data set. And if we save that and then go back to the browser and reload it. Take a little while and there is all the data as this massive multi-dimensional array. And if we right click and view source we can see the array more easily. There it is there is the date, air temperature, all the values for air temperature.

And if we scroll down, we will see all the different values. It's a very long array. So if we go back to the editing program, this very short script sorts out the filtering of a single year. But there are ten years to process. And before inserting the data into the database, we need to make sure that the format of each file is the same. And the bad news is that the format suddenly changes in 2007. So let's open 2007.txt. As before, the first line consists of text headers, but there are fewer of them, and they're in a different order.

Fortunately, the columns of data are still separated by tabs, and the date and time are still treated as a single column. So all we need to do is find out the index of air temperature, barometric pressure, and wind speed. The index for air temperature is one. Barometric pressure is two. And then wind speed is along here, it's the last one, it's index number seven. So we need to go to our processing script.

And we need to also the values of W, A, B, which are our indexes. So we use the conditional operator, or the turnery operator. So, year is less than 2007. If that is true, we will use one. If it's not true, we'll use seven. And it's the same way here. Just add that, year less than 2007.

We'll use four alternatively we'll use one. I've messed up my question mark there. Let's just quickly add those in. So this works rather like an if/else statement. If year is less than 2007 the value of W will be one. If that's not less than 2007 then it will change to seven. So to process all ten years we can change this first line here, line two, into a for loop. So, four and then we begin with year is 2001, year is less than or equal to 2010 and then increment the year.

And we don't need the printer at the bottom, so we'll get rid of that line. And replace it with our closing brace. Now let's just tidy that up. So this script will extract all the data that we want from the ten years of files. The next thing we need to do is to insert it into the database, and we'll tackle that next.

Show transcript

This video is part of

Image for Code Clinic: PHP
Code Clinic: PHP

41 video lessons · 3236 viewers

David Powers

Expand all | Collapse all
  1. 8m 49s
    1. Welcome
      1m 18s
    2. What you should know before watching this course
      4m 40s
    3. Using the exercise files
      1m 56s
    4. Getting the most from Code Clinic
  2. 1h 34m
    1. Introducing Lake Pend Oreille
      5m 4s
    2. Overview of my solution
      3m 22s
    3. Setting up the database
      4m 55s
    4. Retrieving the full-year records
      5m 20s
    5. Processing the full-year records
      10m 49s
    6. Priming the date_recorded column
      7m 47s
    7. Inserting the full-year data into the database
      6m 48s
    8. Processing individual days
      6m 54s
    9. Retrieving a year's data a day at a time
      7m 41s
    10. Keeping the data up to date
      4m 27s
    11. Calculating the mean and median values
      7m 21s
    12. Creating a web service
      6m 21s
    13. Getting the required data
      6m 50s
    14. Building the web service response
      10m 38s
  3. 49m 47s
    1. Identify the image subset
      3m 14s
    2. Overview of my solution
      3m 19s
    3. Setting up the files
      2m 43s
    4. Calculating the scaling ratio
      6m 28s
    5. Using a custom class to scale images
      6m 25s
    6. Generating the thumbnails
      7m 49s
    7. Using subimage-search in ImageMagick
      4m 32s
    8. Analyzing the image statistics
      4m 45s
    9. Finding the cropped images
      7m 13s
    10. Running the script and displaying the results
      3m 19s
  4. 39m 3s
    1. A classic CS interview question
      2m 4s
    2. Overview of my solution
      3m 48s
    3. Finding all possible combinations
      6m 26s
    4. Detecting horizontal attacks programmatically
      4m 8s
    5. Implementing the checkLayout() function
      6m 25s
    6. Rotating the chessboard
      4m 59s
    7. Eliminating duplicate solutions
      6m 34s
    8. Displaying the unique solutions
      4m 39s
  5. 21m 33s
    1. Build a musical instrument using mouse movements NEW
      1m 35s
    2. Overview of my solution NEW
      2m 1s
    3. Using the Web Audio API NEW
      2m 47s
    4. Creating the tone generator NEW
      7m 39s
    5. Controlling the frequency and volume NEW
      7m 31s

Start learning today

Get unlimited access to all courses for just $25/month.

Become a member
Sometimes @lynda teaches me how to use a program and sometimes Lynda.com changes my life forever. @JosefShutter
@lynda lynda.com is an absolute life saver when it comes to learning todays software. Definitely recommend it! #higherlearning @Michael_Caraway
@lynda The best thing online! Your database of courses is great! To the mark and very helpful. Thanks! @ru22more
Got to create something yesterday I never thought I could do. #thanks @lynda @Ngventurella
I really do love @lynda as a learning platform. Never stop learning and developing, it’s probably our greatest gift as a species! @soundslikedavid
@lynda just subscribed to lynda.com all I can say its brilliant join now trust me @ButchSamurai
@lynda is an awesome resource. The membership is priceless if you take advantage of it. @diabetic_techie
One of the best decision I made this year. Buy a 1yr subscription to @lynda @cybercaptive
guys lynda.com (@lynda) is the best. So far I’ve learned Java, principles of OO programming, and now learning about MS project @lucasmitchell
Signed back up to @lynda dot com. I’ve missed it!! Proper geeking out right now! #timetolearn #geek @JayGodbold
Share a link to this course

What are exercise files?

Exercise files are the same files the author uses in the course. Save time by downloading the author's files instead of setting up your own files, and learn by following along with the instructor.

Can I take this course without the exercise files?

Yes! If you decide you would like the exercise files later, you can upgrade to a premium account any time.

Become a member Download sample files See plans and pricing

Please wait... please wait ...
Upgrade to get access to exercise files.

Exercise files video

How to use exercise files.

Learn by watching, listening, and doing, Exercise files are the same files the author uses in the course, so you can download them and follow along Premium memberships include access to all exercise files in the library.

Exercise files

Exercise files video

How to use exercise files.

For additional information on downloading and using exercise files, watch our instructional video or read the instructions in the FAQ .

This course includes free exercise files, so you can practice while you watch the course. To access all the exercise files in our library, become a Premium Member.

Are you sure you want to mark all the videos in this course as unwatched?

This will not affect your course history, your reports, or your certificates of completion for this course.

Mark all as unwatched Cancel


You have completed Code Clinic: PHP.

Return to your organization's learning portal to continue training, or close this page.

Become a member to add this course to a playlist

Join today and get unlimited access to the entire library of video courses—and create as many playlists as you like.

Get started

Already a member ?

Become a member to like this course.

Join today and get unlimited access to the entire library of video courses.

Get started

Already a member?

Exercise files

Learn by watching, listening, and doing! Exercise files are the same files the author uses in the course, so you can download them and follow along. Exercise files are available with all Premium memberships. Learn more

Get started

Already a Premium member?

Exercise files video

How to use exercise files.

Ask a question

Thanks for contacting us.
You’ll hear from our Customer Service team within 24 hours.

Please enter the text shown below:

The classic layout automatically defaults to the latest Flash Player.

To choose a different player, hold the cursor over your name at the top right of any lynda.com page and choose Site preferences from the dropdown menu.

Continue to classic layout Stay on new layout
Exercise files

Access exercise files from a button right under the course name.

Mark videos as unwatched

Remove icons showing you already watched videos if you want to start over.

Control your viewing experience

Make the video wide, narrow, full-screen, or pop the player out of the page into its own window.

Interactive transcripts

Click on text in the transcript to jump to that spot in the video. As the video plays, the relevant spot in the transcript will be highlighted.

Are you sure you want to delete this note?


Your file was successfully uploaded.

Thanks for signing up.

We’ll send you a confirmation email shortly.

Sign up and receive emails about lynda.com and our online training library:

Here’s our privacy policy with more details about how we handle your information.

Keep up with news, tips, and latest courses with emails from lynda.com.

Sign up and receive emails about lynda.com and our online training library:

Here’s our privacy policy with more details about how we handle your information.

submit Lightbox submit clicked
Terms and conditions of use

We've updated our terms and conditions (now called terms of service).Go
Review and accept our updated terms of service.