Questions tagged [csv]

21566 questions
1

votes
1

answer
228

Views

Reading text file with special characters using d3.request

I have a file.txt that I need to get to my script and parse via d3.request. The content of file is encoded with windows-1250 encoding and has extra lines to be deleted, so that only lines starting with 'Date' and '2017' should pass. So far I have been using cli solution to grep text file (removing e...
mtx
1

votes
1

answer
487

Views

create schema for imported csv in mongodb

i have a csv that looks like this: Number,Date,ClientName,ClientSurname,ClientAge,Option1,Option2 1,01/01/2015,John,Foo,24,30, using mongoimport i get { '_id':ObjectID(''), 'Number':1, 'date':'01/01/2015', 'ClientName':'John', 'ClientSurname':'Foo', 'ClientAge':24, 'Option1':30, 'Option2': '', } w...
doe
1

votes
0

answer
123

Views

Create a dictionary from a csv with variable length entries

I have a CSV with with 8 keys. Each key contains a list of a variable length for its values. I used csv.DictWriter to create the csv, now I want to read in the csv into a dictionary. I've poured through the docs on csv.DictReader and csv.reader and this is what I've come up with: def ReadCSVasDic...
Drew Wilkins
1

votes
2

answer
106

Views

Insert into second column after specific id into csv

I have a table named A with 1000 rows Like: id | name ----+------ 1 | a 2 | b 3 | c 4 | d and so on until id=1000 Now I want to export this table A as CSV which i am able to do. But I want data in 2 columns like data till id=499 should print in 1st column and remaining data should prin...
Rids
1

votes
0

answer
63

Views

How to use comments in csv table to get a text classification?

I have several statements if people like a product or not I have anonymized the comments, and used other names. As the classification problem is very similar to those of movie reviews, I relied totally on the tutorial from http://www.nltk.org/book/ch06.html. Section 1.3. from nltk.corpus import movi...
smurfit89
1

votes
1

answer
137

Views

write csv rows one by one rather than all at once in python3 using writer.writerows(reader)

I am saving a csv file after skipping the rows before the header: print (latest_file) with open(latest_file, 'r',encoding='utf8') as infile: reader = csv.reader(infile) for row in reader: #look for if row[0] == 'date/time': print (row) break else: print('{} not found'.format('name')) with open('C:/...
1

votes
0

answer
249

Views

Big CSV to pickled dict

I have a 1 GB CSV file like this (16-char-max-string;int) hello;1798732 you;1762 ... that is slow to be parsed into a dict, that's why I would like to do it once, save it as pickled file (or any other serialization method), and then reopen this already-processed file for further use. I tried:...
Basj
0

votes
5

answer
28

Views

Extracting .txt data into a meaningful table for analysis using Python (or R)

I'm working on an engineering project in which I'm using machine performance data from archives. The machine produces one data set approximately every 5s and this data is then available date-wise in a number of .txt files with each file containing data in the following format. The data shown below...
kd68
-1

votes
3

answer
49

Views

Fast reading large table

I have csv file structured as below: 1,0,2.2,0,0,0,0,1.2,0 0,1,2,4,0,1,0.2,0.1,0 0,0,2,3,0,0,0,1.2,2.1 0,0,0,1,2,1,0,0.2,0.1 0,0,1,0,2.1,0.1,0,1.2 0,0,2,3,0,1.1,0.1,1.2 0,0.2,0,1.2,2,0,3.2,0 0,0,1.2,0,2.2,0,0,1.1 but with 10k columns and 10k rows. I want to read it in such a way that in the result i...
Adam Stawarek
1

votes
1

answer
249

Views

Writing flatten data.frame in .csv from a Shiny app

I have a problem with managing a dynamic data.frame created in my Shiny app. Problem is, since it is handled through the server.R, I can't find a way to access it, flatten it into a vector then write it after my other variables (which are all text/numeric inputs). Working app can be accessed here :...
Samuel Tremblay
1

votes
1

answer
276

Views

Getting KeyError while executing executemany command in python

Python code import pymysql import xlrd import re import os import csv conn = pymysql.connect(host='', user='', password='', db='', charset='utf8mb4', cursorclass=pymysql.cursors.DictCursor) sql = 'INSERT INTO `webscrap_wunderground` (Date,MeanTemp,MaxTemp,MinTemp,HeatingDegree,DewPoint,MeanHumidity,...
sarika
1

votes
1

answer
325

Views

How to remove unwanted double quotes when converting CSV to JSON in angular

In my CSV file, I have this: id | level | date 1 | 2 | 01-01-2017 2 | 4 | 01-01-2017 After using the code from here, I am able to get this string on my console: id,level,date 1,2,'01-01-2017' 2,4,'01-01-2017' The code I used for the results above to be converted to JSON is from here: var l...
icedmilocode
1

votes
1

answer
296

Views

Import CSV file with extension yii2-csv-importer

I'm not sure how to use the extension 'yii2-csv-importer'. I need to import a CSV file that is in the app/web/uploads/mod-key.csv directory My table/model (BASE) and csv file have the following structure: 'id' 'date' 'Pan 'key' 'goal' 'accomplished' 'Weight' 'points' 'update' How would my ACTION loo...
gugoan
1

votes
0

answer
261

Views

How to print csv report in browser odoo?

I am trying to print some fields of account.invoice and account.invoice.line Firstly I added the button for printing csv formatted report. this is i trying in code def print_csv(self): lines = [] lines.append((['Product Name','Price Unit','Quantity'])) for rec in self.invoice_line_ids: lin...
aslamsha22
1

votes
0

answer
84

Views

Removing blank lines in csv output on Windows 10 without an error

I'm using Scrapy 1.4.0 and Python 3.6.3. When I run 'scrapy crawl -o items.csv', every other line of the csv file is blank. While I did find a solution here, it generates the following error: 2017-12-31 18:39:48 [scrapy.utils.signal] ERROR: Error caught on signal handler: Traceback (most recent...
Dan
1

votes
1

answer
95

Views

Skip the first line of a CSV file in Yii2

public function actionUpload(){ $model = new CsvForm(); if($model->load(Yii::$app->request->post())){ $file = UploadedFile::getInstance($model,'file'); $filename = 'Data.'.$file->extension; $upload = $file->saveAs('uploads/'.$filename); if($upload){ define('CSV_PATH','uploads/'); $csv_file = CSV_PAT...
Janardhan Ram
1

votes
2

answer
780

Views

mysq - Use variable (date) when SELECT * INTO OUTFILE (on file name)

Good afternoon, I'm using INTO OUTFILE to create a CSV file with the information of a SELECT. I need to use variables to personalize the name of the CSV. For example, I want to add the date NOW(). This is the idea (but its not working): SET @outpath = 'out/'; SET @outfile = (SELECT NOW()); SET @oute...
Nicolau Roca
1

votes
2

answer
222

Views

Merge CSV in neo4j avoiding duplicates

I am Working with neo4j and an example Data-set of Movies. https://neo4j.com/developer/example-data/ I now want to import and update my database with an CSV file of grouplens https://grouplens.org/datasets/movielens/ If the movie written in the csv-file is already in the database i want to update...
user9167430
1

votes
0

answer
190

Views

how to continue with uploading a csv file to postgres using flask+panda

i am new to python and postgres and am trying to build a simple flask web app. a user can choose a csv file, press upload, and it will create a table in postgres on my server with the table in the csv. a simple copy paste. my html code: upload my python module: from flask import Flask, render_templa...
Tal Alterzon
1

votes
0

answer
258

Views

How do I import multiple excel files and merge them using openpyxl

How do I import muleiple excel files and merge them using openpyxl the code I have so far is this..... from openpyxl import Workbook def Test (filepath, yaml_sheetname): assumed_cell = 'A1' wb = openpyxl.load_workbook('/Users/c1carpenter/Desktop/Test.xlsx') wb = openpyxl.load_workbook('/Users/c1carp...
Carolina Estrella
1

votes
2

answer
213

Views

How to duplicate a python DictReader object?

I'm currently trying to modify a DictReader object to strip all the spaces for every cell in the csv. I have this function: def read_the_csv(input_file): csv_reader = csv.DictReader(input_file) for row in csv_reader: for key, value in row.items(): value.strip() return csv_reader However, the issue w...
user3802348
1

votes
2

answer
72

Views

Combine columns and sort the values before creating a new column

I am making a python script that I want to combine several columns of string data and sort them alphabetically before creating the new column. To simplify my example here is a really simple example of the format of the data I am dealing with: Ingredient 1, Ingredient 2, Ingredient 3 pickles, beef, m...
Marcel
1

votes
1

answer
642

Views

Export to CSV file using 'xlsx' in angular2

I want to export csv file using 'xlsx' npm module but in documentation it doesnt show any direct method for that. I have a json output and i want to export it using 'xlsx' module.Please help me for doing that. I tried this way but not working const CSV_TYPE ='text/csv;charset=utf-8'; const CSV_EXTEN...
user
1

votes
1

answer
65

Views

Pandas and CSV Libraries CSV Manipulation

I am building a simple app. I want some values in my CSV to be updated every 15 minutes. I want this part of my app to run in the background to prevent blocking the interface. I couldn't get it to work the way I want to. My code: # I'm using the pandas, sched and time imports here: #INTERFACE @app....
Anne
1

votes
0

answer
27

Views

How to set a exception in a xpath filter for a similar sounding word?

I'm trying to import a large .csv file (60k entries) in one of my projects by using an importer tool. This tool allows me to filter the entries with the help of a xpath command, which is very useful, because I only need a few of these entries actually. So I'm trying to filter the [title] element by...
B4rT
1

votes
3

answer
181

Views

Python 3 - Load JSON data into my .csv -file

I actually try to write JSON in a data.csv file. I tried following solution from stackoverflow: How do I write a Python dictionary to a csv file? So I come up with these: with open('data/dataGold.csv', 'w') as f: w = csv.DictWriter(f, ['data']['user']['repositories']['nodes'], extrasaction='ignore')...
testElmo
1

votes
1

answer
421

Views

Computing eucledian distance between vectors

I don't understand how the dist() function works that computes the euclidean distance between two vectors. I created the simplest one dimensional vectors and I would like an explanation of the result I am getting. Here is the vector as an CSV file called test.csv: Item Name Attribute Item 1 3...
Dures
1

votes
0

answer
28

Views

Python3 download remote gzip file to csv reader in memory

I'm trying to migrate a project to python3. I'm downloading a gzip file, reading in memory and parsing as a csv reader object. When it comes to looping through the reader, i'm only getting the first character instead of the whole row. Where am i going wrong? response = requests.get(feed_url, stream=...
lennard
1

votes
1

answer
125

Views

Python: write data to CSV from UDP

I used a socket library to receive data from UDP. Data which I send have 32bit width (4 Bytes), My code: with open(csvf, 'w', newline='', encoding='ascii') as csv_handle: csv_writer = csv.writer(csv_handle, delimiter=',') while(True): try: data,addr = s.recvfrom(buf) csv_writer.writerow(data) exce...
kvantum.nuly
1

votes
0

answer
69

Views

python json to csv , code can only parse part of json data

I have the json data want to convert to csv [ { 'status': 'changed', 'dataset': { 'id': '5a4b463c855d783af4f5f695', 'name': 'AE_E', 'label': '1- ADVERSE EVENTS - Not Analyzed' }, 'details': { 'variables': [ { 'variable': { 'id': '5a4b4647855d783b494f9d3f', 'name': 'CPEVENT', 'label': 'CPEVENT' }, 's...
Allen Wei
1

votes
1

answer
549

Views

Read a dsv file with custom delimiter into R

I have a dsv file with the custom delimiter |. I can easily read this into Stata by using the GUI to select a custom delimiter, but all the variants of read.csv that I've tried in R haven't worked. Does anyone know how to how to import a dsv file into R with a custom delimiter?
Obie K
1

votes
0

answer
2.3k

Views

Why is pandas.read_sas() so much slower than pandas.read_csv()?

I'm writing a Python script to do some data work for me and I have a question about performance. I'm running Python 3.6 and the most up-to-date package of pandas. OS is Windows 10. Anyway here's the jist - I'm reading in a SAS table of about 71 million rows and 39 columns in this way: want = [] for...
Nic Payne
1

votes
1

answer
53

Views

Write csv file with non-numeric columns quoted and no row names

I'm trying to write a csv file from a data frame, i.e: Col_A Col_B Col_C Hello World 4 Once More 21 Hi Data 23 So far I use this code: ds = dataf write.csv(ds,'test.csv', row.names = FALSE, quote = c(1,2), sep = ',') However, the result is: Col_A,'Col_B','Col_C'...
João Fernandes
1

votes
0

answer
43

Views

Encoding Error in R command line but not in RStudio

I'm using a Windows 7 machine with R 3.0.0 installed trying to run a script I've written through the R command line. The script takes in a .csv file (created using Save As... in Excel in MS-DOS and UTF-8 format) and applies a number of Dplyr functions to it before writing the results to csv. The scr...
user450889
1

votes
2

answer
73

Views

HTML - Access data from remote online CSV files

I've been trying to implement some HTML code that accesses weather data in historical CSV files online, and perform maths on the data once I selectively extract it. In the past, I've programmed in Python and had no problems doing this by using pycurl.Curl(). HTML is a complete nightmare in comparis...
poopipeboy
1

votes
0

answer
59

Views

mongodb - strategy from having relational DB CSV dump imported to highly denormalised mongodb documents

We want to migrate data to mongodb using a CSV files dump created out of a teradata. Data needs to be refreshed in mongodb every night from a fresh teradata csv dump Approach we are thinking of is: Get the CSV files exported out of relational db. They are going to be very similar to table structure...
Mohsin Hakak
1

votes
2

answer
2.9k

Views

EmptyDataError: No columns to parse from file

Currently I am getting the below Error and I am tried out the below posts: Solution 1 Solution 2 But I am not able to get the error resolved. My python code is as below: import pandas as pd testdata = pd.read_csv(file_name, header=None, delim_whitespace=True) I tried to print the value in testdata b...
Haritha
1

votes
1

answer
810

Views

export from mysql database to excel file

I'm trying to export table from MySQL database to excel file using PHP it's working correctly but Arabic language words displays as strange letters I tried everyhting in the internet but nothing worked you can see my code and tell me what I can do pleeeeeeeeease I'm working on it for 3 days now :( t...
Abanob William
1

votes
1

answer
122

Views

Python CSV: Append to COLUMN, duplicate entries

I'm trying to write to a CSV file, every new entry to a column, but every time the CSV file gets duplicate entries. I'd want to create the 1, 2, 3 rows only if they do not exist, to be the first, and then add data to column 1, 2 and 3. I am having the current code: import _csv myData = [[1, 2, 3], [...
Dav Marika
1

votes
2

answer
60

Views

How can I read the csv file?

I have a data, which is seperated by ',' and I have 7 lines of that. I am trying to read the data seperated by comma into 2d vector like this: ifstream input; input.open('wordplate.csv'); vector data; string line; while(getline(input,line)){ cout

View additional questions