41

I am trying to retrieve all items in a dynamodb table using a query. Below is my code:

import boto.dynamodb2
from boto.dynamodb2.table import Table
from time import sleep

c    = boto.dynamodb2.connect_to_region(aws_access_key_id="XXX",aws_secret_access_key="XXX",region_name="us-west-2")

tab  = Table("rip.irc",connection=c)

x    = tab.query()

for i in x:
    print i
    sleep(1)

However, I recieve the following error:

ValidationException: ValidationException: 400 Bad Request
{'message': 'Conditions can be of length 1 or 2 only', '__type': 'com.amazon.coral.validate#ValidationException'}

The code I have is pretty straightforward and out of the boto dynamodb2 docs, so I am not sure why I am getting the above error. Any insights would be appreciated (new to this and a bit lost). Thanks

EDIT: I have both an hash key and a range key. I am able to query by specific hash keys. For example,

x = tab.query(hash__eq="2014-01-20 05:06:29")

How can I retrieve all items though?

6 Answers 6

65

Ahh ok, figured it out. If anyone needs:

You can't use the query method on a table without specifying a specific hash key. The method to use instead is scan. So if I replace:

x    = tab.query()

with

x    = tab.scan()

I get all the items in my table.

Sign up to request clarification or add additional context in comments.

3 Comments

This is not a solution, you are not gonna use scan for your app logic, it is very costly! You should rethink your indexes and use query or get_item !
scan is a reasonable solution for small tables. Source: docs.aws.amazon.com/amazondynamodb/latest/developerguide/…
.scan() also does not automatically return all elements in a table due to pagination of the table, there is a 1mB limit to each response. After that, you will need to make subsequent scans with the "LastEvaluatedKey" attribute
14

.scan() does not automatically return all elements of a table due to pagination of the table. There is a 1Mb max response limit Dynamodb Max response limit

Here is a recursive implementation of the boto3 scan:

import boto3
dynamo = boto3.resource('dynamodb')


def scanRecursive(tableName, **kwargs):
        """
        NOTE: Anytime you are filtering by a specific equivalency attribute such as id, name 
        or date equal to ... etc., you should consider using a query not scan

        kwargs are any parameters you want to pass to the scan operation
        """
        dbTable = dynamo.Table(tableName)
        response = dbTable.scan(**kwargs)
        if kwargs.get('Select')=="COUNT":
            return response.get('Count')
        data = response.get('Items')
        while 'LastEvaluatedKey' in response:
            response = kwargs.get('table').scan(ExclusiveStartKey=response['LastEvaluatedKey'], **kwargs)
            data.extend(response['Items'])
        return data

1 Comment

Why not just use the pre-built scan paginator?
13

I'm on groovy but it's gonna drop you a hint. Error :

{'message': 'Conditions can be of length 1 or 2 only'}

is telling you that your key condition can be length 1 -> hashKey only, or length 2 -> hashKey + rangeKey. All what's in a query on a top of keys will provoke this error. The reason of this error is: you are trying to run search query but using key condition query. You have to add separate filterCondition to perform your query. My code

    String keyQuery = " hashKey = :hashKey and rangeKey between :start and :end "
    queryRequest.setKeyConditionExpression(keyQuery)// define key query
    String filterExpression = " yourParam = :yourParam "
    queryRequest.setFilterExpression(filterExpression)// define filter expression
    queryRequest.setExpressionAttributeValues(expressionAttributeValues)
    queryRequest.setSelect('ALL_ATTRIBUTES')
    QueryResult queryResult = client.query(queryRequest)

1 Comment

Perfect! Thanks @Вадим, you saved me a lot of time!
3

I ran into this error when I was misusing KeyConditionExpression instead of FilterExpression when querying a dynamodb table.

KeyConditionExpression should only be used with partition key or sort key values. FilterExpression should be used when you want filter your results even more.

However do note, using FilterExpression uses the same reads as it would without, because it performs the query based on the keyConditionExpression. It then removes items from the results based on your FilterExpression.

Source Working with Queries

Comments

3

Inspired by the other answers and comments. If all records are needed use scan instead of query (it requires key filters).

ref: scan, paginator('scan')

import boto3
from boto3.dynamodb.types import TypeDeserializer

session = boto3.Session()
client = session.client("dynamodb")

paginator = client.get_paginator('scan')

response_iterator = paginator.paginate(
    TableName=table_name
)

data = []
for page in response_iterator:
    for item in page['Items']:
        # convert to python types
        deserializer = TypeDeserializer()
        python_data = {k: deserializer.deserialize(v) for k, v in item.items()}

        # append to list
        data.append(python_data['data'])

Comments

-5

This is how I do a query if someone still needs a solution:

def method_name(a, b)
    results = self.query(
      key_condition_expression: '#T = :t',
      filter_expression: 'contains(#S, :s)',
      expression_attribute_names: {
        '#T' => 'your_table_field_name',
        '#S' => 'your_table_field_name'
      },
      expression_attribute_values: {
        ':t' => a,
        ':s' => b
      }
    )
    results
  end

1 Comment

This does not answer the question in a useful way. why do you believe this is the answer? how does it work? Simply telling someone to change their code without any context or meaning does not help them learn what they did wrong.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.