1,585 questions
2
votes
0
answers
53
views
Why does a DELETE with a JOIN on partitioned columns in BigQuery cost more than dropping specific partitions?
I have a large BigQuery table, big_table, around 5 TB in size.
It is partitioned by the column partition_date, which has about 2000 distinct values.
I also have a smaller table, small_table, which ...
- 21
0
votes
0
answers
31
views
How to programmatically handle container partition redistribution in GridDB cluster after node failure?
Question
GridDB Container Partition Recovery After Node Failure
I'm working with a 3-node GridDB cluster and need to implement automatic recovery logic when one node fails. My application creates ...
1
vote
1
answer
86
views
When a new broker is added, how does kafka moves partitions for rebalancing? [closed]
I am new to Kafka and if I understand correctly we need to
Specify upfront the number of partitions
We can increase partitions if required but we can't delete them (at the risk of losing data)
The ...
- 4,529
0
votes
1
answer
42
views
KNIME: how to access the configure window in the new version
I'm following a course on Udemy where the instructor is using an older version than me. When he right-click on "scatter plot" or "table partitioner" he gets an option to open a ...
- 101
1
vote
2
answers
59
views
In Kafka, if a message is manually sent to a specific partition, will subsequent messages with the same key go the same partition?
Consider the following sequence:
Message1 with key "A" is sent to a specific partition of a topic:
kafkaProducer.send("my-topic", 0, "A", "Message content");
...
- 13
2
votes
4
answers
200
views
Ruby group array of ints by total without going over
In Ruby, I want to group an array of integers by a running total without the total going over a certain number. While keeping the integers in order.
I have this array of numbers from zero to one ...
- 2,138
0
votes
1
answer
239
views
In esp32,Build is failing because of partition table
I am using ESP-IDF v5.4.1.I have a project with custom partition table.When I building that project.When I run idf.py build command,build is getting failed.
Logs of failure:
[110/1369] Generating ../.....
1
vote
2
answers
94
views
How to Assign RowNumber for a Couple Unique Values [duplicate]
I have a query that joins multiple tables. The result of the query is something similar to this temptable:
DROP TABLE IF EXISTS #temptable
CREATE TABLE #temptable
(
[rowNum] int,
[Id] int,
...
- 693
0
votes
2
answers
65
views
SQL row-wise comparison within group
I am trying to compute the number of events that take place within 30 minutes of one another (variable n_within_30) on the same date and in the same location.
I have tried to partition and group by ...
0
votes
0
answers
43
views
Rabbimq cluster_partition_handling
I'm administering 3 clusters RabbitMQ versions 3.10 and 3.11 in a production environment. I wasn't the one who installed those clusters.
I had to solve a production incident, involving partition. ...
- 171
0
votes
0
answers
44
views
Insert overwrite multiple partitions in an external Hive table
I am trying to overwrite multiple partitions in a large table. Basically I have my main external S3 table sandbox, partitioned by part:
scala> q("select * from sandbox")
+---+-------------...
- 85
0
votes
0
answers
36
views
PR-10 The flash partitioning MUST allow later addition of new partitions (prplfoundation)
To support the above feature I found out a way to update GPT via approach mentioned by me in the code section. But in the approach if there is any power failure during sgdisk command execution then ...
5
votes
1
answer
112
views
getting every combination of every combination of strings in array
In lua I want to get every possible combination of every possible combination of whole strings in array... Let me explain what I mean by that.
Let's say that we got an array C = { "a", "...
- 53
1
vote
2
answers
89
views
SQL Server sum over partition by
I have a fact table and I am using sum over partition by on the column vacant. I want to sum vacant when vacant is above 0. I have the following query:
SELECT
f.[D_Building_Id],
f.[...
- 21
0
votes
1
answer
50
views
Event Ordering Across Tables in Kafka
In Kafka, it's generally recommended to map one table to one topic. However, what if the order of events across multiple tables is important for a specific key, such as CompanyId?
For example, ...
- 108
1
vote
0
answers
32
views
Load all subpartitions within a specific partition dynamically AWS athena present in s3
CREATE EXTERNAL TABLE testpart
(id bigint,
eventday bigint,
eventhour bigint
PARTITIONED BY (eventday smallint,eventhour bigint)
ROW FORMAT SERDE'org.apache.hadoop.hive.ql.io.parquet.serde....
1
vote
1
answer
87
views
How does MySQL handle SELECT queries on partitioned tables?
When querying a non-partitioned table, the query optimizer can leverage indexes for sorting and limit the data read based on the LIMIT clause. For example, in a non-partitioned table my_table with a ...
- 11
2
votes
0
answers
39
views
How to fix the directory name format from 'key=value' to 'value' generated by PySpark 'partition_by' function?
I'm trying to customize directory names generated by PySpark using partition_by on these two columns (date and customer),
Dataframe is similar to this:
+----------+--------+
|date |customer|
+----...
-2
votes
1
answer
54
views
Maximize sum of last three remaining elements [closed]
You are given an array containing n elements, where n is odd. In one operation, you
can select an index i, such that 1 ≤ i < n and remove ai and ai+1 from the array
and concatenate the remaining ...
1
vote
1
answer
90
views
How to calculate moving average over two columns in SQL
I am trying to calculate a moving average for a team score, but this should include both the points scored when it is the away team and home team.
For example, I have the following table Match:
...
0
votes
1
answer
49
views
Oracle SQL - Get rows where it meets a percent allocation
I am working on selecting participants for a program based on certain criteria and after meeting the criteria, get x% (say 60%) from a specific department A, and the remaining 20% from another ...
- 161
2
votes
3
answers
101
views
Create random partition inside a pandas dataframe and create a field that identifies partitions
I have created the following pandas dataframe:
ds = {'col1':[1.0,2.1,2.2,3.1,41,5.2,5.0,6.1,7.1,10]}
df = pd.DataFrame(data=ds)
The dataframe looks like this:
print(df)
col1
0 1.0
1 2.1
2 2....
- 1,762
0
votes
0
answers
72
views
How do I prevent duplicate rows in Postgresql while using COUNT and partitions?
first time question asker here and pretty new to SQL. I'm doing a project in postgresql and pgadmin 4 (if that matters) and I'm getting duplicates in my 'title' and 'title_mo_rentals' fields when ...
- 1
0
votes
1
answer
80
views
Selecting a Common ID where that ID Matches Certain Criteria
In the screenshot, Id is the primary key. There could be many of the same Participant__c values in this result set, each with a different Id.
What I'm trying to do is write a query that will return ...
- 10.1k
0
votes
1
answer
42
views
LEFT JOIN on ROW_NUMBER OVER PARTITION
I have two tables :
1.Actions : Contains multiple actions performed on orderIDs on specific dates. For each orderID, there are multiple rows, each representing a different action performed on a ...
2
votes
1
answer
83
views
Find the partition separator for a given Linux block device
I'm writing a script that will partition several disks on a Linux system. These disks can be all sorts of block devices. I need to find a way to reliably create the names of the partitions that will ...
- 43
0
votes
0
answers
156
views
BigQuery - How to efficiently join 2 large tables every 5 minutes without incurring a large cost
This question is a follow up question to Why is there such a difference in performance between these two queries?.
I am having difficulty finding an efficient way to frequently query and join two ...
- 151
0
votes
1
answer
93
views
Why is one stopping condition for quicksort's partitioning step with two pointers L and R, while (L <= R)? Can it be while (L < R)?
I'm trying to understand the difference between these two conditions for the partition step of quicksort. While writing this out, I'm capitalizing single letter variable names so it's more readable.
...
1
vote
1
answer
101
views
Partition a list into subsets of a different length
I have a set of different numbers, e.g.
set = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110].
Now I want to partition this list into sublists which are not longer than 5.
I know how I can partition ...
- 11
1
vote
1
answer
186
views
Having Trouble Implementing Partition-key-based Multi-Tenancy and Search with Milvus
I am attempting to implement Partition-key-based multi-tenancy, but I was not able to find any code implementation tutorial for me to follow along.
Here is the code I have so far. However, I am having ...
- 193
0
votes
0
answers
144
views
Partitions for Delta Lake External Tables in Snowflake
i am trying to create partitions for External Tables of type Delta Lake. Unfortunately the Delta Lake file system does not have a partition implemented neither, which means no partition is mentioned ...
-1
votes
1
answer
115
views
ESP32 max partition table size doesn't seem to matter
I'm programming the ESP32 S2 WROOM with PlatformIO with a custom partition table that allows me to store as much data as possible in the app0 partition, and quite a lot of user settings in the nvs ...
- 1
0
votes
1
answer
140
views
Cannot use IN function for Partition Filter of a View Table in BigQuery
so I have a view table named table_1 and it has a partition field called business_date. Thie table_1 is constructed by using a raw table named data_mart.customer_raw.
There are a lot business needs ...
- 339
0
votes
1
answer
156
views
COUNT(DISTINCT ...) OVER (PARTITION BY ...) in Postgrsql
I have the following query:
SELECT s.first_field,
s.second_field,
t.third_field,
s.fourth_field,
ROUND(
(SUM(mt.volume) * 100.0 /
SUM(mt....
- 3
0
votes
1
answer
67
views
Is it possible to partition an Athena table using a variable which is timestamp datatype
I am trying to partition an athena table by a timestamp variable called 'forecast_datetime'. To use this variable as a partition I have made it 'string', with following values:
'2007-12-10-11-00-00', '...
- 597
1
vote
1
answer
302
views
Parted in scripted mode can increase but not decrease partition size
I am trying to use parted in scripted mode to resize a partition.
When I use the command to increase the partition size it works but when I use the same command but simply change the sector size to ...
- 583
0
votes
0
answers
56
views
PySpark cumulative sum of contiguous rows over partition
I have a table that holds reports that come in for different entities, and the state the entity is in at the time of the report. It is important to know the total duration of each occurrence of an ...
- 355
7
votes
0
answers
131
views
Number of partitions of (a,b) into k distinct parts which sum up to (a,b) [closed]
Problem set
This is somewhat a generalization of the famous partition of integer n into k parts.
Given two integers a,b I need to find the number of partitions into k distinct parts that sum up to (a,...
- 119
0
votes
2
answers
604
views
Batch file with disk part
I've been trying to create a batch file that calls out diskpart commands and then exits back to the batch file. The diskpart commands increase the size of the recovery partition.
Here is the current ...
0
votes
0
answers
60
views
Reuse partition across multiple queries with different QUALIFYs
Is it possible to accomplish the following or something similar?
In particular, what I would like to accomplish is the reuse of some_partition across different queries (query1, query2) each using ...
- 2,985
0
votes
2
answers
99
views
Column comparison between rows of the same group used in Partition By clause
I want to compare same columns of rows falling into the same group created by partition by clause in SQL. My table has below data:
Id Serial_Number Last_update_date Flag
1 3344 07-...
0
votes
1
answer
52
views
Create bigger DATETIME intervals from little intervals and aggregate values with a SELECT query
I have a table in MySQL with stock data, that has the columns:
time
open
high
low
close
volume
I get an entry every 5 minutes. How do I get these values in an hourly interval instead of a 5 minute ...
- 1
0
votes
1
answer
432
views
How to set up retention period in dbt model if it run next day need to run dag without fail
We did dbt configuration for one of the model with incremental strategy and unique key and again customer wants to setup retention period for 7day did as per requirement and logic Is working only once ...
0
votes
1
answer
47
views
Return rows with last updated date for different days
Assume this is my PySpark dataframe, ordered by ("ID", "updated_at"):
ID
updated_at
stock_date
row_num
a1
2024-03-25T20:52:36
2024-03-25
1
a1
2024-03-26T11:23:48
2024-03-26
2
a1
...
0
votes
0
answers
93
views
Split array strictly in two parts where left half elements are smaller right half elements
I need to divide the array A [n] into two sub-arrays A1, A2 based on the middle value midval, in a manner that all the elements of A1 (or denoted left) are smaller than the midval and all the elements ...
- 69
1
vote
1
answer
153
views
Incredibly slow count(*) on MariaDB / MySQL
Running EXPLAIN SELECT COUNT(*) FROM activities shows it plans to use the secondary index with and only one column int and keylen of 5. Action takes anywhere from 2 minutes to 6 minutes, sporadically
...
0
votes
1
answer
197
views
How to update the values in a subpartition table in oracle which has a lot of data?
This is my exisitng table:
PARTITION BY RANGE ("D0_CRD_RPT_DT") INTERVAL (NUMTOYMINTERVAL(1, 'MONTH'))
SUBPARTITION BY LIST ("TA_SRC_STM_CD","TA_OBJ_TP_CD")
...
0
votes
1
answer
52
views
How to GROUP BY based on newly created MAX() column
My table:
Name
ID
Status
Roger Collins
904
3
Roger John Horspool
915
3
Roger John Shippey
932
3
Roger John Shippey & T.C. Rowell
5341
2
Roger John Shippey & T.C. Rowell
5341
3
Due to poor ...
- 73
0
votes
1
answer
199
views
Does ClickHouse support partitioning like traditional RDBMS do, and if so how can I implement it?
What are the feasibility and implementation methods of partitioning data in ClickHouse?
What are the traditional RDBMS?
In ClickHouse, how feasible is data partitioning, and what implementation ...
0
votes
0
answers
45
views
Is Multi level partitioning possible in SQL Server [duplicate]
I have a table with 7 columns in it. Table consist of 30 millon records. I already did range partition on date column. Now I want to further create sub-partition or partition by partition. I'm ...
