Mongodb bulk write error

Question

I'm executing bulk write

bulk = new_packets.initialize_ordered_bulk_op()

bulk.insert(packet)

output = bulk.execute()

and getting an error that I interpret to mean that packet is not a dict. However, I do know that it is a dict. What could be the problem?

Here is the error:

    BulkWriteError                            Traceback (most recent call last)
    <ipython-input-311-93f16dce5714> in <module>()
          2 
          3 bulk.insert(packet)
    ----> 4 output = bulk.execute()

    C:\Users\e306654\AppData\Local\Continuum\Anaconda\lib\site-packages\pymongo\bulk.pyc in execute(self, write_concern)
583         if write_concern and not isinstance(write_concern, dict):
584             raise TypeError('write_concern must be an instance of dict')
    --> 585         return self.__bulk.execute(write_concern)

    C:\Users\e306654\AppData\Local\Continuum\Anaconda\lib\site-packages\pymongo\bulk.pyc in execute(self, write_concern)
429             self.execute_no_results(generator)
430         elif client.max_wire_version > 1:
    --> 431             return self.execute_command(generator, write_concern)
432         else:
433             return self.execute_legacy(generator, write_concern)

    C:\Users\e306654\AppData\Local\Continuum\Anaconda\lib\site-packages\pymongo\bulk.pyc in execute_command(self, generator, write_concern)
296                 full_result['writeErrors'].sort(
297                     key=lambda error: error['index'])
    --> 298             raise BulkWriteError(full_result)
299         return full_result
300 

    BulkWriteError: batch op errors occurred

Ok, the problem was that i was assigning _id explicitly and it turns out that the string was larger than 12-byte limit, my bad. — David Makovoz
– David Makovoz, Commented May 20, 2015 at 17:24
PyMongo acknowledges this is a common issue in their docs and offers insight as to why this occurs. — tutuDajuju
– tutuDajuju, Commented Dec 13, 2015 at 9:01

Dima Lituiev · Accepted Answer · 2020-03-30 23:24:54Z

33

It can be many reasons...
the best is that you try...catch... the exception and check in the errors

from pymongo.errors import BulkWriteError
try:
    bulk.execute()
except BulkWriteError as bwe:
    print(bwe.details)
    #you can also take this component and do more analysis
    #werrors = bwe.details['writeErrors']
    raise

edited Mar 30, 2020 at 23:24

Dima Lituiev

13.2k10 gold badges47 silver badges63 bronze badges

answered Sep 7, 2016 at 3:19

Samer Aamar

1,4082 gold badges17 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

David Makovoz · Accepted Answer · 2015-05-20 17:25:14Z

20

Ok, the problem was that i was assigning _id explicitly and it turns out that the string was larger than 12-byte limit, my bad.

answered May 20, 2015 at 17:25

David Makovoz

1,9383 gold badges18 silver badges34 bronze badges

1 Comment

Dennis Golomazov Over a year ago

wow, I faced exactly the same problem, thank you for the solution!

Miguel Angel · Accepted Answer · 2015-07-10 19:53:06Z

17

You should check 2 things:

Duplicates, if you are defining your own key.
Be able to manage custom types, In my case I was trying to pass a hash type object that was not able to be converted into a valid objectId, and that was leading me to the first point and I felt into a vicious circle (I solve it converting myObject to string.

Inserting one by one will give you the idea what is happening.

answered Jul 10, 2015 at 19:53

Miguel Angel

4493 silver badges6 bronze badges

Comments

bwl1289 · Accepted Answer · 2021-03-02 02:14:14Z

In addition to the above, check your unique indexes. If you're bulk inserting and have specified an index that doesn't exist in your data, you will get this error.

For example, I had accidentally specified name as a unique index, and the data I was inserting had no keys called name. After the first entry is inserted into mongo, it will throw this error because you're technically inserting another document with a unique name of null.

Here's a part of my model definition where I'm declaring a unique index:

self.conn[self.collection_name].create_index(
            [("name", ASCENDING)],
            unique=True,
        )

And here are the details of the error being thrown:

{'writeErrors': [{'index': 1, 'code': 11000, 'keyPattern': {'name': 1},
'keyValue': {'name': None}, 'errmsg': 'E11000 duplicate key error collection:
troposphere.temp index: name_1 dup key: { name: null }'
...

more resources: MongoDB E11000 duplicate key error

Niharika · Accepted Answer · 2022-05-03 09:15:51Z

0

I was trying to insert two documents with the same "_id" and other keys. Solution:

insert different "_id" s for different documents. OR
remove the "_id" and you get a randomized one.

answered May 3, 2022 at 9:15

Niharika

11 bronze badge

1 Comment

Jeremy Caney Over a year ago

This is covered by @Miguel Angel's answer from seven years ago.

Lamak · Accepted Answer · 2020-06-29 08:39:07Z

-1

Try using debugger, it should gives you errmsg with exact error, and op object was trying to insert.

answered Jun 29, 2020 at 8:39

Lamak

1012 silver badges9 bronze badges

Collectives™ on Stack Overflow

Mongodb bulk write error

6 Answers 6

Comments

1 Comment

Comments

Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

Comments

1 Comment

Comments

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related