Easy-to-follow video tutorials help you learn software, creative, and business skills.Become a member

Working with appended tabular data

From: Interactive Data Visualization with Processing

Video: Working with appended tabular data

If you want to do any data visualization in Processing, the single most important thing to do is to actually get your data into Processing, and that can also be an unusually challenging procedure, because Processing doesn't really have any built-in functions for doing this. On the other hand, a number of people have developed methods to facilitate the integration of external data files into Processing, and I'm going to show you one of them, probably the simplest, and it's for reading and external spreadsheet file saved as a tab-separated values, or TSV, file and using those values in Processing, by using a table class developed by Ben Fry, who is one of the cofounders of Processing.

Working with appended tabular data

If you want to do any data visualization in Processing, the single most important thing to do is to actually get your data into Processing, and that can also be an unusually challenging procedure, because Processing doesn't really have any built-in functions for doing this. On the other hand, a number of people have developed methods to facilitate the integration of external data files into Processing, and I'm going to show you one of them, probably the simplest, and it's for reading and external spreadsheet file saved as a tab-separated values, or TSV, file and using those values in Processing, by using a table class developed by Ben Fry, who is one of the cofounders of Processing.

Now, the good news about version 2.0 of Processing is that the table class will become part of Processing. And it's not there yet, because we're still using an alpha right now, but I can only assume that since it's probably going to be written by the same guy, that it will be very similar to what we have right here. Let's open up our sketch folder, and you see that we've got a few things in there. Number one is the actual Processing sketch, and I can show you that I've got that opened, and that's what that looks like right there. The other one is the table.

That's a class file that contains special instructions for Processing on how to read tabular data. And we're not going to mess with that one. We're just going to say that pretty much, it works. In case you are worried about not be able to find this code again, you should know that Ben Fry's table class is available in the Built-In Examples in Processing; that's what I am showing you right here. The way you get that is by coming up to File, to Examples, and this window pops up. And the one we're looking for is Book/Visualizing Data/ch03-usmap, and then the second one, which is step01_fig1_red_dots, if you click on that, then you see the same table class right there.

What I also have in here is a data folder, and I have two versions of the same data. Now, the reason I did that is because I got some data from Google Correlate that compares states on relative interest in search terms. In fact, I am just going to open that up. I've got it in Excel right now. And here's what we have. We've got Alabama down through Wyoming. We also have Washington D.C. in the mix, and that's important because D.C. tends to function usually in a lot of statistics. We have the state name.

We have the region and the division that there in. Then we have the percentage of adults in that state with degrees, and we have the median age of the population. And so those are a couple of descriptive statistics, but all the rest of that is Google's search terms. And what it is, these signify the relative interest of that state and that search term compared to other states. So what you see here, for instance--let's find a good one. That's video games, and what you find is that Washington D.C. shows a much lower interest in searching on Google for video games than other places.

On the other hand, you find that Hawaii has an unusually high interest in the show Top Chef. And I'll let you make it that what you will. But these are based on search patterns nationally and these data from Google. They're a few months old, but the patterns tend not to change dramatically in short amounts of time. Now, the reason I opened this up in the spreadsheet is because the way we're going to be working with the data files in Processing is, you don't want any of the header information.

Right here I have four rows on the top. The first one numbers the columns starting at 0, because that's how Processing reads things; it started at 0 and then goes on. Then I have the names of the things that people search for. By the way, what I have is the percent, the degree, the age. Video games is a search term, iPhone is a search term, dance, top chef, nfl, nba, mlb for Major League Baseball and then Major League Soccer. By the way, I included two of those because I am from Utah and we love the NBA more than anybody else in the country apparently.

We also love dance the most, and we also love Major League Soccer the most. Anyhow, those are little points of regional pride. But I have to prepare this with this information here, but then you have to delete it to make a very clean file. And so what I have is a second file that has the data but does not have any of those headers, but I like to keep the one with them so I can consult it. Let me recommended that you always do that: keep a copy of the data with the headers with summary statistics, if you want, for your reference purposes, but also have one that's easy to read.

Now the other thing is this is an Excel .xls file, and while there are libraries that make it possible to read Excel files into Processing, I find they are complicated. It's easier to just go with a standard text file. And so what you need to do is you need to save your file as a text file--actually, a tab-separated value. Now, let me show you how this works. You come up to File > Save As. We're going to Enable Saving, and then I have a choice here.

I can do stateData and then say it's Workbook, and then I click on that. I get a whole long list of choices. The one that you want is this one right here, Text (Tab delimited). And when you save that it will come out with a .txt extension at the end, and you just click on it and you change it manually to TSV. You would think that the CSV, the comma- separated value, would be a good one, but I encountered some really significant problems in reading that data into Processing, so I just say stick with the TSV for now and then hopefully the Table class will get improved and be a little more flexible when the full version of Processing 2.0 comes around.

So I am just going hit Cancel because I have that one already. By the way, if your data is in Google Docs spreadsheet, you can do Export or Download as Text and it saves it as a TVS file anyhow. So, I am just going to go back to my Processing sketch, and here you see my information. I've got a little palette here. And then this is the class. Table is the class, and then stateData is the object that I'm going to create. That's going to be sort of my Data variable.

I'm also creating a variable called rowCount that will count as it goes through the file. I have the setup. I've got a window of 600 x 200 pixels. Then I take my stateData object and I read the data into, by saying New-- we have to do a New when you're dealing with the classes--and then Table is the name of the class and then statesData.tsv in quotes and parentheses. That's the name of the file, and it needs to be in the data folder, in the sketch folder.

I take the variable rowCount, and now I initialize it by using one of the Table functions, and that's getRowCount. So I do stateData, because that's now the name of the object in I am dealing with, .getRowCount and then empty parentheses, and that will tell me how many rows there are in the data. And in fact, I've got this little thing here beneath that to say print line and tell me how many rows there are. I know there are 51 because, well, there should be 51; we'll see if it works right. Beneath that, I have just a little bit of code to draw.

I've got a background for my palette. I've turned on anti-aliasing. Then I have another palette color for some dots that I'd be drawing, but I am making them somewhat transparent. That's the 160. That's the alpha, about halfway between the 0 and 255. And I've also turned off the outlines. Then I have a for loop that does the important sort of heavy lifting here. And what it says is just go into the data file, start at row 0. And remember, I got rid of all the headers in this, so row 0 is a data row.

And it says and then go through one at a time until you get to the end. And what we want to do is we want to pull out the state name. We're just telling it that that's what's in the first one. We're also going to get age. We're going to get degrees. I am just getting those two variables out right now. The other just so going to be there, and what I am going to do is I am going to create a little scatter plot of median age in the state to the percentage of adults with a degree. And what I'm doing down under the ellipse, where I am multiplying things by 12 and 3, that's just a very rough attempt of getting them to spread out across.

You'll see that there is actually big gap on the left, and I can adjust for that manually. But right now this is just a simple version. And also, because once it gets drawn, it's not going to change, or at this point it's not dynamic, I am using the noLoop command to tell Processing it doesn't have to keep going through the draw loop; just do once and we're done. And let's press Ctrl+R on the PC or Command+R on the Mac and see what we get. We've got two things. Number one, take a look down at the bottom of the console, and you'll see that it says rowCount=51.

That confirms that Processing was able to read 51 rows of data from my sheet, which is right because we have the 50 states and we have Washington D.C. And then what you have in the top is an unlabeled, but you know, slightly picturesque scatter plot that shows the association of median age. That's what goes across the bottom, and that one that's way down at the left is Utah. We've got a median age of only 28 compared to most other states. And then going up is the percentage of the population with college degrees.

And so you can see, if you take out Utah and you take out the other outlier that we've got, there's not a real strong association between these two things. Nevertheless, the purpose of this exercise was not so much to produce this one graph; it was to show you how you can read data into Processing. And the easiest way to do this is to save a very clean TSV file from a spreadsheet with no headers on it and use Ben Fry's excellent Table class to import the data into your sketch.

Show transcript

This video is part of

Image for Interactive Data Visualization with Processing
Interactive Data Visualization with Processing

72 video lessons · 12722 viewers

Barton Poulson
Author

 
Expand all | Collapse all
  1. 3m 16s
    1. Welcome
      58s
    2. What you should know
      1m 22s
    3. Using the exercise files
      56s
  2. 11m 51s
    1. Overview of data visualization
      11m 51s
  3. 11m 53s
    1. Installing Processing
      3m 38s
    2. Overview of Processing
      4m 5s
    3. Exploring libraries
      4m 10s
  4. 1h 1m
    1. Basic setup
      7m 31s
    2. Drawing points
      4m 37s
    3. Drawing lines
      5m 6s
    4. Drawing ellipses and circles
      5m 24s
    5. Drawing arcs
      6m 54s
    6. Drawing rectangles and squares
      4m 58s
    7. Drawing quadrangles
      3m 25s
    8. Drawing triangles
      2m 55s
    9. Drawing polygons
      3m 37s
    10. Drawing simple curves
      4m 54s
    11. Drawing complex curves
      6m 46s
    12. Drawing Bézier curves
      5m 38s
  5. 54m 3s
    1. Introduction to variables
      10m 44s
    2. Understanding variable scope
      6m 53s
    3. Modifying variables
      9m 8s
    4. Creating arrays
      9m 53s
    5. Modifying arrays
      6m 37s
    6. Creating strings
      7m 3s
    7. Modifying strings
      3m 45s
  6. 1h 2m
    1. Incorporating randomness
      7m 59s
    2. Using Perlin noise
      4m 24s
    3. Shuffling with Java
      3m 30s
    4. Specifying line attributes
      8m 2s
    5. Changing placement modes
      5m 45s
    6. Understanding color attributes and functions
      4m 16s
    7. Exploring color spaces
      7m 44s
    8. Using color palettes
      7m 5s
    9. Transforming the grid
      8m 38s
    10. Exploring the attribute matrix
      5m 33s
  7. 52m 7s
    1. Building code blocks
      5m 57s
    2. Writing a while loop
      3m 52s
    3. Using for loops
      5m 35s
    4. Creating conditionals
      14m 50s
    5. Working with easing
      10m 51s
    6. Creating spirals
      11m 2s
  8. 18m 55s
    1. Mouse tracking
      3m 54s
    2. Hovering and clicking
      11m 16s
    3. Understanding keyboard interaction
      3m 45s
  9. 27m 32s
    1. Specifying fonts
      6m 43s
    2. Using images
      5m 51s
    3. Playing a video loop
      6m 20s
    4. Exporting video
      3m 47s
    5. Adding sound
      4m 51s
  10. 20m 49s
    1. Creating functions
      11m 48s
    2. Creating classes and objects
      9m 1s
  11. 31m 10s
    1. Using embedded data
      5m 26s
    2. Working with appended text data
      6m 4s
    3. Working with appended tabular data
      10m 26s
    4. Reading XML data
      9m 14s
  12. 48m 17s
    1. Generating dot plots
      11m 11s
    2. Building scatter plots
      10m 0s
    3. Making line plots
      9m 55s
    4. Creating bar charts
      9m 12s
    5. Checking out examples of maps, hierarchies, and networks
      7m 59s
  13. 20m 57s
    1. Introducing some principles of 2D design
      13m 44s
    2. Understanding color theory
      7m 13s
  14. 24m 46s
    1. Interacting with zooming, rotating, and sliding
      6m 26s
    2. Implementing slicing
      6m 47s
    3. Using rollovers
      5m 58s
    4. Introducing the GUI libraries
      5m 35s
  15. 10m 35s
    1. Sharing via OpenProcessing and other sites
      3m 19s
    2. Saving as a desktop application
      2m 42s
    3. Saving as JavaScript
      1m 47s
    4. Saving as an Android application
      2m 47s
  16. 2m 38s
    1. Where to go from here
      2m 38s

Start learning today

Get unlimited access to all courses for just $25/month.

Become a member
Sometimes @lynda teaches me how to use a program and sometimes Lynda.com changes my life forever. @JosefShutter
@lynda lynda.com is an absolute life saver when it comes to learning todays software. Definitely recommend it! #higherlearning @Michael_Caraway
@lynda The best thing online! Your database of courses is great! To the mark and very helpful. Thanks! @ru22more
Got to create something yesterday I never thought I could do. #thanks @lynda @Ngventurella
I really do love @lynda as a learning platform. Never stop learning and developing, it’s probably our greatest gift as a species! @soundslikedavid
@lynda just subscribed to lynda.com all I can say its brilliant join now trust me @ButchSamurai
@lynda is an awesome resource. The membership is priceless if you take advantage of it. @diabetic_techie
One of the best decision I made this year. Buy a 1yr subscription to @lynda @cybercaptive
guys lynda.com (@lynda) is the best. So far I’ve learned Java, principles of OO programming, and now learning about MS project @lucasmitchell
Signed back up to @lynda dot com. I’ve missed it!! Proper geeking out right now! #timetolearn #geek @JayGodbold
Share a link to this course

What are exercise files?

Exercise files are the same files the author uses in the course. Save time by downloading the author's files instead of setting up your own files, and learn by following along with the instructor.

Can I take this course without the exercise files?

Yes! If you decide you would like the exercise files later, you can upgrade to a premium account any time.

Become a member Download sample files See plans and pricing

Please wait... please wait ...
Upgrade to get access to exercise files.

Exercise files video

How to use exercise files.

Learn by watching, listening, and doing, Exercise files are the same files the author uses in the course, so you can download them and follow along Premium memberships include access to all exercise files in the library.


Exercise files

Exercise files video

How to use exercise files.

For additional information on downloading and using exercise files, watch our instructional video or read the instructions in the FAQ .

This course includes free exercise files, so you can practice while you watch the course. To access all the exercise files in our library, become a Premium Member.

Are you sure you want to mark all the videos in this course as unwatched?

This will not affect your course history, your reports, or your certificates of completion for this course.


Mark all as unwatched Cancel

Congratulations

You have completed Interactive Data Visualization with Processing.

Return to your organization's learning portal to continue training, or close this page.


OK
Become a member to add this course to a playlist

Join today and get unlimited access to the entire library of video courses—and create as many playlists as you like.

Get started

Already a member ?

Become a member to like this course.

Join today and get unlimited access to the entire library of video courses.

Get started

Already a member?

Exercise files

Learn by watching, listening, and doing! Exercise files are the same files the author uses in the course, so you can download them and follow along. Exercise files are available with all Premium memberships. Learn more

Get started

Already a Premium member?

Exercise files video

How to use exercise files.

Ask a question

Thanks for contacting us.
You’ll hear from our Customer Service team within 24 hours.

Please enter the text shown below:

The classic layout automatically defaults to the latest Flash Player.

To choose a different player, hold the cursor over your name at the top right of any lynda.com page and choose Site preferences from the dropdown menu.

Continue to classic layout Stay on new layout
Exercise files

Access exercise files from a button right under the course name.

Mark videos as unwatched

Remove icons showing you already watched videos if you want to start over.

Control your viewing experience

Make the video wide, narrow, full-screen, or pop the player out of the page into its own window.

Interactive transcripts

Click on text in the transcript to jump to that spot in the video. As the video plays, the relevant spot in the transcript will be highlighted.

Are you sure you want to delete this note?

No

Your file was successfully uploaded.

Thanks for signing up.

We’ll send you a confirmation email shortly.


Sign up and receive emails about lynda.com and our online training library:

Here’s our privacy policy with more details about how we handle your information.

Keep up with news, tips, and latest courses with emails from lynda.com.

Sign up and receive emails about lynda.com and our online training library:

Here’s our privacy policy with more details about how we handle your information.

   
submit Lightbox submit clicked
Terms and conditions of use

We've updated our terms and conditions (now called terms of service).Go
Review and accept our updated terms of service.