Bug #372

CSV export data integrity issues

Added by Amgine na over 1 year ago. Updated 10 months ago.

Status:Resolved Start date:02/11/2012
Priority:Urgent Due date:
Assignee:- % Done:

70%

Category:development Spent time: -
Target version:1.1.0
Difficulty:average

Description

Rows 1 and 2 of CSV output are both header rows.

missing data

  1. The CSV export contains less rows than reported in the Drupal instance
  2. lat, lon are missing
  3. geofield missing
  4. UUIDs don't match imported values

occupations-missing-from-export.txt (1.8 kB) Andrew Mallis, 02/13/2012 02:57 am

occupations-missing-from-export.txt (2 kB) Inga Jensen, 04/10/2012 09:32 pm


Related issues

related to #OWSdirectory - Bug #371: CSV export reports issues and successful export Rejected 02/11/2012

History

Updated by Andrew Mallis over 1 year ago

  • File occupations-missing-from-export.txt added
  • Subject changed from CSV data issue to CSV export data integrity issues
  • Category set to development
  • Assignee set to Andrew Mallis
  • Priority changed from Normal to High
  • Target version set to 0.6.0

duplicate header rows

I thought this might be related to the views filter being exposed as a block, combined with the "Parent sort" option in the CSV download, which attempts to re-order content before export based on said filters.

I disabled the context which renders the views filter on /occupations, unchecked "parent sort" and downloaded a .csv that stil contained 2 headers.

Also unchecked "Make first row a list of column headers" which (surprise) renders no headers.

missing data

I compared the CSV to the data import based on occupation name to confirm 101 missing occupations. See attached file.
I can't discern a pattern here. I thought there may be escape characters or something, but I didn't see anything initially jarring.

Geofield was missing from the fieldset; I added that back in.

UUIDs don't correspond to imported values

Incidentally, I noticed that UUID isn't an INT. Basically what we imported there is bunk. The UUID ends up like: d425968a-3e69-c8b4-615b-3412836001dd when I regenerated missing UUIDs.
If we could get all our records exported as a CSV, it would be a simple matter to update the original google doc so we can match and update.

Updated by Andrew Mallis about 1 year ago

  • Assignee deleted (Andrew Mallis)
  • % Done changed from 0 to 70

So the missing record issue is all that's left it seems. Unassigning myself from this, since I haven't been able to solve the matter.

Updated by Inga Jensen about 1 year ago

I just downloaded a CSV which had 1325 rows compared to 1416 in the Directory...91 missing.

When this issue initially came up, I created a notepad with the 45 missing Occupations listed (attached).

Also of note: there are two Hamilton's in the Directory, in both exports, the one in Canada was exported and the one in NY was not.

Occupy Los Angeles, Phoenix were not exported the first time...but they are present in the export file this time.

Andrew, this does not appear to have fixed itself ;)

Updated by Andrew Mallis 11 months ago

  • Priority changed from High to Urgent
  • Target version changed from 0.6.0 to 1.1.0

Updated by Joel Farris 10 months ago

  • Status changed from New to Resolved
  • Difficulty set to average

Testing this with @inga and @andrew just now, the number of records in the db and the number of records in the exported .csv are the same. Marking this one as fixed.

Also available in: Atom PDF