Questions tagged [pymongo]

1

votes
2

answer
687

Views

MongoDB : Can't insert twice the same document

On my pymongo code, inserting twice the same doc raises an error : document = {'auteur' : 'romain', 'text' : 'premier post', 'tag' : 'test2', 'date' : datetime.datetime.utcnow()} collection.insert_one(document) collection.insert_one(document) raises : DuplicateKeyError: E11000 duplicate key error c...
Romain Jouin
1

votes
1

answer
158

Views

Authentication failed error in Python script to connect MongoDB server using pymodm

The MongoDB server is hosted in a docker container. Created a super user with root permission to login using username and password. Using pymodm to connect to MongoDB server from Python script. I'm able to connect to the docker hosted MongoDB server from the shell as well as the Compass tool. The ef...
Sarabjeet Singh
1

votes
3

answer
4.7k

Views

How to install 3. Party library into anaconda if it is not in conda list

I have a general problem about module importation. Thank you very much. The situation is the following: I have a python compressed package *.tar.gz This package can not be found in conda list if I uncompressed it and use 'python setup.py install' package do would be installed into system python nam...
Hello lad
1

votes
2

answer
1.8k

Views

cursor.next() always returns “java.util.NoSuchElementException”

I am writing a program in java which reads query results from MongoDB. Below is the code (which gives an exception): if(output != null){ BasicDBObject whereQuery = new BasicDBObject(); DBCursor cursor = null; for (DBObject obj : output.results()) { String UserID = obj.get('UserID').toString(); where...
rakemen
1

votes
1

answer
66

Views

Pymongo: How can I make this, “double query”?

Hello and happy new year to everybody. This question is because I have a problem and I don't know what to look for as a solution. I think that aggregation is no good here. I am a novice level programmer starting with python 3.6 and pymongo 3.4 to handle Mongodb. I have 50M documents inside a 'lostIt...
Deus
1

votes
1

answer
226

Views

Pymongo aggregation - passing python list for aggregation

Here is my attempt at performing the aggregation (day-wise) based on timestamp if all the elements are hardcoded inside the query. pipe = [ { '$match': { 'cid': ObjectId('57fe39972b8dbc1387b20913') } }, { '$project': { 'animal_dog': '$animal.dog', 'animal_dog_tail': '$animal.dog.tail', 'animal_cat...
elevenmac
1

votes
1

answer
53

Views

Mongodb arithmetic expression of elements inside an array

{ '_id' : ObjectId('5a4d0712368ff072a56a66e8'), 'a' : { 'nodes' : [ { 'b' : 1 }, { 'c' : { 'nodes' : [ { 'd' : 2 }, { 'e' : 2 } ] } } ] }, 'r' : 1 } { '_id' : ObjectId('5a4d0712368ff072a56a66e9'), 'a' : { 'nodes' : [ { 'b' : 4 }, { 'c' : { 'nodes' : [ { 'e' : 9 } ] } } ] }, 'r' : 2 } Above are the s...
daemon24
1

votes
0

answer
312

Views

Using pyMongo and APscheduler with Gunicorn --preload option

I start gunicorn with 3 workers (--workers 3) to launch a flask app and jobs configured with APscheduler. In order to launch my jobs just once (and not once per worker) I use the --preload option. However, my issue is that I also use pyMongo which does not work safe with the gunicorn --preload optio...
Benjamin
1

votes
2

answer
369

Views

Add a new field with large number of rows to existing collection in Mongodb

I have an existing collection with close to 1 million number of docs, now I'd like to append a new field data to this collection. (I'm using PyMongo) For example, my existing collection db.actions looks like: ... {'_id':12345, 'A': 'apple', 'B': 'milk'} {'_id':12346, 'A': 'pear', 'B': 'juice'} ... N...
Sam
1

votes
1

answer
22

Views

Is there any way to decrease the time it takes for a pymongo.errors.AutoReconnect to occur?

I'm using a 3-node MongoDB replica set and connecting to it using Pymongo v3.3.1. In testing the handling of errors like AutoReconnect and ServerSelectionTimeout and such I find that I can't (safely/reliably) control how long it takes to raise an AutoReconnect exception. If I instantiate MongoClient...
John Carrell
1

votes
0

answer
116

Views

Run independent, Parallel or Multithread to increase speed

I have 2 separate collections. Each has information of hotels in the world , each provided by different companies, but both contain same information, each collection have information like gps, name , country, city , email , fax and tel the problem is name(gps , info and ...) are changed, I wrote a s...
koorosh safeashrafi
1

votes
0

answer
54

Views

How can I update and delete record in mongo using pymongo

I am trying to do insert/update/delete using pymongo. my insert works, but update and delete does not success def update(self,test1): print 'hi' client=MongoClient('localhost',27017) db=client.test collection=db.test1 # result = collection.test1.update_one({'col1': 1}, {'$set': {'col2': 'name'}})...
Aasif
1

votes
0

answer
1.1k

Views

Can I get all documents of mongodb collection in list?

I am using mongodb for data storage. I am using pymongo for fetching data from mongo server. I want to get all documents from each collection into list. Following are my database statistics shown by mongodb compass: Does it get all documents in list? My computer is having 32 GB ram. curs=db.collecti...
Avinash Magar
1

votes
0

answer
101

Views

Mongodb: Modify all elements in the specified array field using pymongo

The following is an example from Mongodb Documentation to explain how to modify all elements in the specified array field. Consider a collection test with the following documents: { '_id' : 1, 'grades' : [ 85, 82, 80 ] } { '_id' : 2, 'grades' : [ 88, 90, 92 ] } { '_id' : 3, 'grades' : [ 85, 100, 90...
Jianli Cheng
1

votes
0

answer
23

Views

InvalidBSON Exception while reading documents from mongodb

I am getting InvalidBSON exception at third line while reading documents from mongodb using pymongo. I want to continue iteration even after exception anyone know how to do it? curs=coll.find() documents=[] for doc in curs: documents.append(doc)
Avinash Magar
1

votes
1

answer
1.4k

Views

Connect to mLab

I tried connecting to mLab using Python but I'm still having trouble with connection. import pymongo from pymongo import MongoClient class TensorDB(object): ''' def __init__(self, ip = 'ds125058.mlab.com', port = 25058, db_name = 'mkdnb', user_name = 'manal',password = '******', ... ): client = Mong...
Ruba Suliman
1

votes
1

answer
320

Views

MongoDB/pymongo calculate difference of two times in minutes

I am trying to calculate the difference between two timestamps in mongodb. With MySQL it can easily be achieved using this query SELECT ROUND((TIME_TO_SEC(NOW()) - TIME_TO_SEC(lastseen))/60) AS minutes where lastseen is a timestamp column. Here is my schema: { '_id' : ObjectId('5aa329cb0b717a0f63...
tablebubble
1

votes
3

answer
106

Views

MongoDB: Exclude fields that start with a typical string

My Dataset looks like that: { 'A_asdsd' : '1', 'A_fghf' : '1', 'B_tzzz' : '1', 'B_ghh' : '1',.... } How do I have to specify my find() Function, that all fields that start with A_ are excluded? (There are thousand of them).
Covo
1

votes
0

answer
91

Views

Insert field value as “number” type

I have a collection which has some existing data and there is a particular field which is causing a little problem for me. The field value is like this, 'user_id' : 106962662714214320000 So, when I get the type of this field, > typeof db.cbRel.findOne().user_id number Now, I am inserting data to mon...
Luv33preet
1

votes
2

answer
259

Views

Check if item field exist in pymongo database using scrapy

I'm trying to check if item ['email'] already exists in database, if it does not exist then insert into pymongo database. i do not want duplicate emails within the pymongo database. however i get this ValueError: dictionary update sequence element #0 has length 17; 2 is required this is what i hav...
JimmySmith
1

votes
0

answer
59

Views

Remove duplicates values in aggregation with limit in Mongo

I have a dataset (an example): {u'geometry': {u'type': u'Point', u'coordinates': [151.5162, -9.44365]}, u'_id': ObjectId('5ad70f71f2119236741ffb39'), u'type': u'Feature', u'properties': {u'POS_ID': u'592795', u'STATUS': u'0', u'TIMESTAMP': u'2013-12-31 18:52:00.000', u'MMSI': u'636015725'}} {u'geome...
Antonis
1

votes
1

answer
107

Views

Pymongo insert and query reference

I have 2 table and I want to insert a reference to the first table into the second table. This is the table I want to reference, named player self.db.insert('player', {'char_id': char_info.char_id, 'name': char_info.name, } I tried following the official docs and do this: self.db.insert('admin', {'...
Trax
1

votes
0

answer
27

Views

How to Parrallelize Read operation in MongoDB in Spark?

I am using mongodb/mongo-hadoop (https://github.com/mongodb/mongo-hadoop/wiki/Spark-Usage#python-example) but was confused on how I can do parallel read operations. By parallel read operations means, do concurrent read operation on my MongoDB index. I have indexed my DB based on Timestamp, and wan...
Rohit Ranjan
1

votes
0

answer
48

Views

PyMongo - “language override unsupported: C++” when creating index

I have the collection posts that contains posts that look something like this { '_id': '5ae37fd270f3e72399988198', 'moderator': { 'flagged': false, 'reviewed': true, 'pending': false, 'time': '2018-04-27 20:34:38.099000', 'account': 'samhamou' }, 'author': 'cryptohazard', 'permlink': 'security-enhan...
Esoemah
1

votes
1

answer
191

Views

Server error on Flask-Pymongo app deployed to Heroku… what am I missing?

I'll try and provide as much information as possible here. The app works locally just fine. The app is deployed successfully, and I can get to the login screen... but when I log in, I then see the following error: Internal Server Error The server encountered an internal error and was unable to compl...
acd37
1

votes
0

answer
45

Views

Flask Pymongo Objectid link not working

I am tring to access a new document from a mongo database collection named games by the _id. But for example if I access localhost:5000/solutie/5ae71f3e8e442b090e4c313bit is giving me the error: ValueError: View function did not return a response so it doesn't go through the if and I think I should...
Ax M
1

votes
1

answer
47

Views

Can't get Pyramid to recognize Mongo

I am trying to get the Pyramid Web framework to handle a request using Mongo but I am a relative newbie to both. I cannot get my view to recognize a database attached to a request. In development.ini: ### # configure mongodb ### mongo_uri = mongodb://localhost:27017/nomad The __init__.py imports an...
RubyNoob
1

votes
0

answer
77

Views

How to use PyMongo to connect to a (CA) SSL enabled replica set

I use the following code with PyMongo to connect to a replica set (according to the official documentation): from pymongo import MongoClient client = MongoClient( ['1.1.1.1:27017', '2.2.2.2:27017', '3.3.3.3:27017'], username='user', password='password', ) I can use the following code to connect to c...
Dinal24
1

votes
0

answer
44

Views

python motor create_index doesnt delete raw after expired time

I have the following test in my project: async def test_expire_at( aiohttp_client, mongo_collection): await mongo_collection.delete_many({}) await mongo_collection.create_index([('expireAt', 1)], expireAfterSeconds=0) expire = datetime.utcnow() + timedelta(seconds=5) await mongo_collection.update_on...
Alexander
1

votes
0

answer
48

Views

how to parse data of all collections using pymongo

enter image description hereI have a mongodb connection(posted in image),which is having a collection of say some 20 files. i would like to read/load each collection from python and do the necessary computing. Please find the sample below what i am trying to do. import pandas as pd import json fr...
sri lakshmi
1

votes
0

answer
18

Views

python script runs MongoClient but not with Apache

I've installed all the requirements to work with pymongo+apache+python... but for some reason when i run this from the webbrowser it reports in the error log but it works fine from the shell with a normal ./script.py If i comment the from pymongo.... it runs ok with the browser. Any clues? [client...
Tony
1

votes
0

answer
62

Views

MongoEngine version for MongoDB 3.2.20

Our MongoDB version is to be upgraded to 3.2 from 3.0. This matrix shows we need PyMongo version to be 3.2 for working with mongodb 3.2. We will be upgrading PyMongo to 3.2. But not sure which MongoEngine version to move to. MongoEngine site says 3.2+ is not supported yet. Any inputs on which Mongo...
1

votes
0

answer
29

Views

Migrate records from elasticsearch to mongodb

I want to migrate all the records from elasticsearch (nearly half a million records) to a mongodb collection. I want to know the best way to do it.
siva prasad
1

votes
1

answer
75

Views

Python:when I try to insert JSON file into Mongodb,if the json file is not valid its throwing an error.how to catch the error

Hi I am trying to insert json files into mongoDb.I have written the code and is working fine for me. import sys, json, pymongo,glob from pymongo import MongoClient from bson import json_util from multiprocessing.sharedctypes import template from json.decoder import JSONDecodeError from itertool...
Solomartian
1

votes
1

answer
22

Views

Load inconsistent data in pymongo

I am working with pymongo and am wanting to ensure that data saved can be loaded even if additional data elements have been added to the schema. I have used this for classes that don't need to have the information processed before assigning it to class attributes: class MyClass(object): def __init_...
jonyfries
1

votes
1

answer
200

Views

Convert mongodump BSON files to Parquet

I am trying to convert the data from a mongoDB dump to a parquet format. I am using python, pymongo and a shell script for that. When I run the shell script to dumb the data I got a lot of BSON files, I did not find a way to direct convert BSON -> Parquet, so I am trying to do BSON -> JSON -> Parque...
tanaka
1

votes
0

answer
656

Views

mongodb update query to append a data to already existing field [duplicate]

This question already has an answer here: Updating nested documents in mongodb 2 answers Update MongoDB field using value of another field 7 answers Append item to MongoDB document array in PyMongo without re-insertion 3 answers For a web application form i'm trying to add data to mongo collectio...
Rachel
1

votes
0

answer
30

Views

Nested insertion in MongoDB

Consider the following structure in MongoDB database in a collection called programs { '_id': 'Program123', 'versions' : { 'v1' : 'First Version', 'v1.4' : 'First version with changes' } } } Suppose that I want to add a new version field v2.0 : 'Improved version' without having to repeat the inform...
bolzano
1

votes
1

answer
25

Views

Opposite single/double quotes using pymongo command

I am trying to take user input, create a URI, and add it with a collection in Pymongo, but whenever I try to do this, the format gets messed up and I cant figure out how to fix it. When running the line: print(db.command('create', 'storage', someStorage={ 'URI': {FS_URI}})) where 'Storage' is the...
mel
1

votes
0

answer
33

Views

MongoDB - Query evenly distributed number of results based on field values and query limit

Let's say i have a collection with documents like this, with field 'family' and 3 different values (fruit, vegetable, nut). Collection: { 'name': banana, 'family': fruit }, { 'name': apple, 'family': fruit }, { 'name': orange, 'family': fruit }, { 'name': cucumber, 'family': vegetable }, { 'name...

View additional questions