Importing content using feeds
Video: Importing content using feedsOne of the problem domains I encounter often is the need to import content from a data source into Drupal. The United States Department of Energy created the Open Energy Information initiative or OpenEI as a platform to connect the world's energy data. Licensed under a Creative Commons Public Domain one of the many collections of open data they publish is a database of wind farms in the United States. As this module facilitates the creation of a database of wind farms, I'm going to use the freely available CSV to import data.
- Creating a custom permission
Viewers: in countries Watching now:
Extend your Drupal 7 sites with custom modules, which allow you to create everything from admin interfaces to forms. Author Jon Peck describes how modules extend your base Drupal installation, then walks through how to write your own module with a practical example featuring geo-positioned alternative energy centers. The course also describes how to control access to site features, create new content types, build forms, understand data persistence, embrace coding standards, and much more.
- Creating your first module
- Interacting with hooks
- Working with permissions and roles
- Controlling access
- Adding a menu item to an admin interface
- Using the Form API (FAPI) to quickly create a form
- Creating custom form validation
- Manually creating a custom content type
- Validating user input
- Importing content using feeds
- Creating a block
- Understanding best practices and coding standards
Importing content using feeds
One of the problem domains I encounter often is the need to import content from a data source into Drupal. The United States Department of Energy created the Open Energy Information initiative or OpenEI as a platform to connect the world's energy data. Licensed under a Creative Commons Public Domain one of the many collections of open data they publish is a database of wind farms in the United States. As this module facilitates the creation of a database of wind farms, I'm going to use the freely available CSV to import data.
I've trimmed the data set a bit to include only fields that are supported in this content type. One of the many advantages of Drupal is the open platform for developing and sharing content, and functionality, and importing data is one of those areas. The Feeds Module imports and aggregates data as nodes, users, taxonomy terms or even simple database records. Out of the box it supports RSS feeds, CSV files, and much more, and these imports can be performed as one-offs or unscheduled regular intervals.
This particular demonstration is not going to require any custom development for the importer which may seem a little counterintuitive given the name of the course. The point is that while I can do something manually, when somebody else has built and maintained a great wheel that tens of thousands of other people use on a daily basis, there's no reason to reinvent it. Feeds does have some requirements. Download and extract the following modules if they're not already installed. Feeds which is the base module, ctools which is required by feeds; it's also a great development library, and job_scheduler which is required by feeds, and facilitates the scheduling of imports.
When they are available go to the Module list, and enable Feeds Admin UI. The required modules will be enabled as needed. Now that feeds is enabled, I need to setup the mechanism for importing data. Go to Structure then Feeds importers. Each Feed importer can be used to define how data is handled upon import, including what factor will be used, mapping, standards, and so forth.
Click Add Importer, for the name I'll use OpenEI Wind Farms, for the description CSV Import of OpenEI Wind Farm Database, click Create. The first thing that needs to be configured is the basic settings which defines general behavior of the importer, click Settings, disable Periodic Import by selecting Off then click Save.
Next, configure the Fetcher which gets the data from a source. There are two options right now; File upload and HTTP Fetcher, switch to a File upload and click Save. Then change the Parser. By default Feeds will parse RSS and Atom Feeds. Switch to the CSV parser, then click Save. By default, the Processor used will be the Node processor which creates and updates nodes from the parsed content.
This is what I want, so no change is needed. Go to the Node Processor Settings, under Content type select Wind Farm, then Save. Finally, define the Mapping. This will define the mapping of the CSV headers to the content type fields. On the left, the Source is the CSV header name, and on the right the Target is the content type field. So we'll add them one at a time. So the first will be Facility Name, I will map it to Title which is the default name of the node title.
In the content type I named this Facility Name, but this interface doesn't check that deeply for, click Add. Next, Facility; I will map this to the Description, just so there's some content. NumberOfUnits, which I will map to Number of Units. Latitude, maps to Latitude.
Longitude, maps to Longitude. And finally, WindTurbineManufacturer. This maps to the Turbine Manufacturer, click Save. That was fairly laborious, wasn't it? Fortunately, this is a one time thing. When I said that custom development wasn't needed I was telling the truth. I'm not writing an importer, however, code will be involved.
One of the tabs on this page is Export. This generates PHP code that can be embedded in a module that will programmatically create the feed importer when that module is enabled. Click on Export now, Select All and Copy. Go to the IDE and open windfarms.module, scroll to the end, then create a doc block. Modules can make hooks available to other modules and feeds is no exception. To provide a default feeds importer, I need to implement hook, feeds, importer, default.
Implements hook_feeds_importer_default(); function windfarms_feeds_importer_default(), which takes no parameters. Feeds exports into a variable called Feeds Importer. Hook_feeds_importer_default returns an array of these feeds. Start with a variable called export with an empty array. $export = array().
Paste the contents of the export in. Then indent it, so it lines up cleanly. Remove the long comment about disabling. At the end, add the feeds importer to export with the key that matches the machine name of the exported feed, which can be found in the feeds importer ID property. In this case, openei_wind_farms; $export 'openei_wind_farms' = $feeds_importer.
Then return $export. Before this will work, Feeds needs to know that this default exists. Ctools has a plug-in API that feeds leverages. So the Wind Farms module needs to let ctools know that it's providing defaults. Implements hook_ctools_plugin_api(); function windfarms_ctools_plugin_api().
The API takes two parameters: the module and the API both strings and both sent to the empty string. ($module = '', $api = ''). Add some logic. If the feeds module and the API question is the Feeds importer default, so if ($module == 'feeds' && $api == 'feeds_importer_ default') then you tell ctools that the API version is 1; return array ('version' => 1). Save then return to the browser.
Go to Structure then Feeds importers, the status is now overridden. If it's not, you'll need to clear the cache, click Revert to go back to the version stored in code. The importer now is using the configuration stored in the module as indicated by the status default. Now that the windfarms module depends on feeds, that information needs to be made available to Drupal, so we can require feeds for enabling Wind Farms. Go back to the IDE, and open the windfarms.info file.
While feeds requires several other modules to function, Wind Farms explicitly depends on feeds, and feeds can handle its own dependencies. Drupal is intelligent enough to enable everything required if you just enable windfarms. Add a line to the end of the info file. Dependencies can contain multiple values so using this standard INI format for arrays, set the key as dependencies and the value to fields, Save. All the groundwork is complete.
The windfarms module will now import, and the configuration is now stored in code. Go back to the browser, then click on the link to the Import page. There should be an option OpenEI Wind Farms, click it now. Go to File, and Choose File. Browse to the location of the windfarms. csv file, then click Open and Import.
The Status Message indicates that 948 nodes are created. Return to the homepage. Returning to the homepage, the latest wind farms to be imported are now shown. Click on one of the imported items. As it looks now, it's not very interesting. In the next segment, I'll add a Google Map of the facility which will be much more useful.
Find answers to the most frequently asked questions about Drupal 7 Custom Module Development .
Here are the FAQs that matched your search "" :
- Q: gmap3_tools is not working the way I expected it to. What version of gmap3_tools should I be using?
- A: Use the free exercise file containing the version of gmap3_tools used for recording; the published version of the module on drupal.org has changed since recording and is not backwards compatible.
- Q: I attempted to run the Drupal site root from the project files, but the site isn't loading. Why not?
- A: The Drupal configuration file in sites/default/settings.php contains database configuration specific to the environment used to record the movie. This may be different than your environment. Edit the file and search for "windfarms" - you may need to change the database host, username, password, db name and port to match your specific environment.
Sorry, there are no matches for your search "" —to search again, type in another word or phrase and click search.