Federated General Assembly » #OWSdirectory

This is awesome. Seems to have worked smoothly.

We'll be merging the spreadsheet you drew from with the others into this (currently empty) master directory:
https://docs.google.com/spreadsheet/ccc?key=0Aldd4WFocycHdDNXQ25ZRGllajNxWkdWQ1FIR210Wnc
which will get populated soon. Fields there aren't final either, though.

The master doc has been updated to reflect what we feel is a reasonable data import.
(There will need to be some light modifications to the Occupation. Will update that task).

We've created 2 records for test import. 1 of them has bad data, just for fun.

Marco, can you please jump in here and work with David on this task?

OK, I applied the patch to allow address integration with feeds (the patch was to the feeds module not to address fields). I've added those mappings and it seems to work great. I'll finish up the rest of the mappings tomorrow evening.

However, there are a few issues regarding what format feeds is expecting the data to be in:

• Country is required. Without country, it can't import the rest of the address fields correctly.
• Country needs to be in the country code two-letter format. (See http://api.drupal.org/api/drupal/includes--iso.inc/function/_country_get_predefined_list/7)
• For some countries, Province/State/Territory also has to be in the two-letter format. Address Field only accepts those for the US, Italy, Brazil, Canada, and Australia. (You can see the options here: http://drupalcode.org/project/addressfield.git/blob/refs/heads/7.x-1.x:/plugins/format/address.inc) For other countries it's a text field, not a select, so it can be anything.

I'm seeing that some rows in the spreadsheet use "(na)" when there is no state/province/terr -- I think those should probably be blanked out.

The rest of the mappings are in place, and everything came in nicely from the template spreadsheet. e.g., http://dev.fga-directory.gotpantheon.com/occupation/occupy-wall-street

The one change I had to make was to add 'US' as the country to Occupy Atlanta.

So assuming you all are cool with conforming to the address limitations I mentioned in my last post here, then we're good to go. If you need more flexibility there because the data is messy, assign back to me and I can write a new plugin to do more complex handling of the country column.

We'll massage the data for you. Great work.

bumping %done down a bit, 'cause it will take a while probably to cycle through the real list(s) once complete.

Having some issues with how feeds imports into taxonomy fields for the twitter hashtags field. Taking this back.

OK, that's working now. Back to Andrew.

Here is the data source:
https://docs.google.com/spreadsheet/ccc?key=0Aldd4WFocycHdEdCVEtBNzFSZ0NUQlpBR1pBN2xON0E

the MASTER sheet should be targeted for import. We need to merge the 4th man data set into that. Forthcoming.

We're cleaning up the spreadsheet a bit, and standardizing our state codes.

I'm hoping to bring the rest of the movement onto this sheet. An inter-occ meeting is in the works for early next week.

Let's get a bit more data into the dev instance, so we can troubleshoot our data format better.

OK, I imported import-master-4th and there are now 1923 items in there. I'm seeing a lot of dupes, etc. but I presume this is still in need of clean-up.

In order to get a successful import, I had to delete about 27 rows that had non-two-letter country abbrevations (e.g., 'global', 'intl', etc.)

Also, because there were no ID's, I had to remove ID from the mapping. If we are intending to do re-imports to update information, I think we'll need that ID, so it's probably worth making sure those get added to the first column. It's OK if they don't get added, but then it will be a one-time import deal.

I've reset all the country names to full spelling.

I've set the state codes for US, Italy, Brazil, Canada, and Australia. Took some doing.

Please run another import.

I'll bring the ID issue up at the data meeting on Wed. No need to implement it unless we agree to that workflow. I'm feeling it's maybe 50-50 right now.

Latest import has been done.

I added another field in the spreadsheet for additional contact emails.
Import wil need some adjustment
Donate information in the data isn't limited to a single URL. I have added a textfield to the content type to handle the next import. Post import, the URL field can be deleted.

Data-wise, a bunch more de-duping has taken place, and occupations that are non-physical have been mostly separated. More to do still, but a new import will make things look better.

OK, I updated the feed mappings to include the new email address field and the new donate field, and ran a new import on the latest spreadsheet.

I didn't delete the donate URL field yet as I think the changes to the export views to use the new fields still need to be made.

Please correct the export views (now a Feature) and delete the donate URL field.

Please re-run an import of the lastest MASTER sheet, which now contains 0 dupes!

Re-ran the import. You probably got a bunch of emails for them from the rules notifications -- sorry about that. I had it set not to send the notification if the node author was anonymous, but that didn't seem to work. I modified it to a positive condition, only emailing if the node author is authenticated. Hope that works next time.

Two rows failed because their donate info value was longer than 128 characters, the max value set for that field. It's a pain to change that value, as it can't be done via the UI unless you delete all content. Do you want that to happen, or can you just update those two rows? (Not sure which two it is, but it would be the two with the longest values.)

Note that I'm currently not importing the ID column because it is missing for some values, and feeds complains if there are missing files for its GUID. Once we do a final import, we can make sure to have that field enabled for the import, assuming all rows have a value for it.

I've added a new field to the Occupation content type that we'll be wanting to import. It's a taxonomy (also added this to the Occupations feature) called occupation_type.

I anticipate we'll be needing to deal with the physical/non-physical occupation question sooner rather than later.

Can you please add this field to the import and run another pass off the MASTER list?

We've spent many hours merging in the information from The Guardian and now how have start dates for a lot of occupations and more specific geo-data.

Ran another import after adding the new taxonomy field.

Still getting two import failures because the donate_info data is too long. I also still have the unique checkbox off for the ID->GUID mapping because there are about 100 rows without an ID.

Here are the import steps:
1) Delete all existing nodes except for Occupy Wall Street. This can be done with VBO, but for now I've been doing it at /devel/php with:

2) Edit occupations content type (/admin/structure/types/manage/occupation/edit) and check the published box so imported occupations are published. You could also do this with VBO.

3) Run the import at import/occupation_csv

4) Edit the occupations content type and uncheck the published box (not needed if you are using VBO to publish the newly imported occupations)

I was getting that same map problem on Friday, but now it's gone. I suspect that it happens when the google api is at its limit. So this won't be a problem after the final import.

Also, I had already added the type taxonomy so I went ahead and removed the duplicate row.

"Also noticed that custom permissions seem to be loosing their values"

I'd open a new ticket for this. Not sure how those were lost, but the solution would be to put these permissions into the occupations feature.

noticed the import-export feature is out of data on DEV, now. Related?

$ drush @fga-directory.dev fr occupations_import_export

or

$ drush @fga-directory.dev fu occupations_import_export

please

Reverted, as the override removed the occupation type field.

Re: Google -- we could try Yahoo Placefinder instead, which allows 5000 geocodes a day as compared to Google's 2500. We'd just need to add an API key here: http://dev.fga-directory.gotpantheon.com/admin/config/content/geocoder and switch the geocoder here: http://dev.fga-directory.gotpantheon.com/admin/structure/types/manage/occupation/fields/field_occupation_geo/edit

Ran our last import (for now)!