632 questions
0
votes
0
answers
40
views
django: Updating a database based on changes on a object or file?
So I am making a website to dynamicaly calculate cost and profit of some crafts(bps) in a game (Eve btw). As I like to do calculations on my own and store my own data, I need to get most of the needed ...
0
votes
0
answers
45
views
How to design a memory-efficient partitioning algorithm for task execution with shared dependencies?
I’m trying to design a partitioning algorithm to scale task execution in a resource-constrained environment. Here’s the situation:
Tasks consume data via a DAL (Data Access Layer), which could be a ...
- 113
1
vote
1
answer
55
views
Efficient way to sum the columns of a Toeplitz/Hankel matrix
There are many repeated values in an (N × N) Toeplitz matrix. Example:
The same is true for its mirror image, the Hankel matrix.
I am trying to find the most efficient way to sum the columns of such ...
- 946
1
vote
1
answer
51
views
How to improve performance on multiplying values in two lists, checking for matching indices
I've created two lists, call them "a" and "b". List a has length 24,480 while b has length 20,116. Every element in both a and b is also a list of length 3. To visualize, each ...
- 11
0
votes
0
answers
91
views
Most efficient way to use a function repeatedly
The goal is to use a formula, for simplicity given as:
def my_formula(x):
return x + 5
, in other functions, in a way that doesn't waste computation time by constantly having to reference back to ...
- 3
0
votes
1
answer
102
views
How to increase VBA Do While loop efficiency?
I am trying to process a column in an excel sheet that contains around 50k rows. The column has text values that need to be evaluated and in case of a correct match, a part of that string needs to be ...
1
vote
1
answer
756
views
Bad query performance postgres 16
We are trying to make the step from postgres 10 to 16, we already tried a while ago to lower versions 12 or 13, but updates were added
in the machine and the poor performance we attribute to those ...
1
vote
1
answer
333
views
Comparing efficiency of two queries
I am learning SQL, and while working through guided examples, I like to attempt writing my own query to answer a question before seeing how the lesson suggests doing it. In this case, I used a ...
- 113
0
votes
0
answers
57
views
What would be the most efficient way to store multiple sets of fixed arrays (std::vector)?
As it stands, I have a game I writing, utilizing PCG (Procedural Content Generation). I was wondering if there was a better way to store the vertices of my polygon with certain library restrictions (...
- 73
0
votes
2
answers
89
views
Optimize a function that analyses a String
I'm currently programming an MCTS AI for a game and I want to optimize the CPU bottlenecks. One method takes about 20% of all processing and returns to what team a piece/base on a position on a 2D ...
- 1
5
votes
1
answer
163
views
R: Efficient Way to partly modify diagonal of matrix
I have a square matrix with dimension ranging from 100x100 to 10,000x10,000. The matrix represents parameter values for a function. I go through a loop where I try various combinations of parameters....
- 165
0
votes
1
answer
60
views
Is it efficient to access/update an array using stored in site options using get_option and update_option in a loop?
I have an array stored in the site options in WordPress which will potentially have 3000 email addresses in it. Every night I need to process this list with some checks on those email addresses and I'...
- 67
0
votes
3
answers
157
views
How to most efficiently repeat the same merge on different columns in R (preferably data.table)
I have program that needs to repeat a merge which uses the same column from Table A over and over, but changes the column from table B. Doing it with a loop that uses the data.table::merge command ...
- 173
-2
votes
1
answer
88
views
Python efficiency - nested loop structure with matrices
I am new to python, but have been coding in MATLAB for years. I've been working to code a specific problem, which requires nested for loops for multiple matrices to solve a series of equations. I knew ...
- 1
2
votes
3
answers
100
views
Looking for a more efficient way to implement this R code if possible
I have the following R code that works but it's quite slow. I want to create a new column based on the values of an existing column in a R dataframe. But there's a catch/complication, I need to access ...
- 23
2
votes
4
answers
666
views
Collatz conjecture with C (range 1-100,000,000) maximum loops
I am making a program that calculates the most loops of the Collatz conjecture between a range of number (1 - 100,000,000) and I need it to be functional in under 4 minutes on a linux system with the ...
- 21
-1
votes
2
answers
151
views
How to be more code efficient on fortran? I solved an exercise and I'm not satisfied with my own answer
I'm studing fortran on my own for a job, I'm very new at it. I tried the following exercise and got a correct answer. However I believe there must be many more processing efficient ways to solve the ...
0
votes
1
answer
672
views
In my Blazor server app I read a file on the server line by line but it takes too long. How can I shorten the time?
In my Blazor server app I want to show the content of a server side file in my razor page. Th file is 5 MB and has appr. 100000 lines. (My app and the file are on the same server, Win2019) I read the ...
- 801
3
votes
1
answer
77
views
What is the most efficient file modification method? Adding many lines to the same file, or reading in, processing, and writing out the file?
I have a perl script that must create a file, write a few hundred lines of content to it, then read all the lines it's written and add lines whenever it finds a match based on a separate configuration ...
- 55
3
votes
1
answer
118
views
How to remove values from a vector which are larger than previous values in the vector
I'm working with a large dataset (400M observations) of cars. Each entry has a VIN number, identifying it as a specific car. I group the dataset by VIN, so each group is 1 car (usually with multiple ...
- 33
0
votes
4
answers
668
views
Efficient maping of large pandas dataframe (by index)
I'm currently optimising my code and I have found bottle neck.
I have dataframe df with column 'Numbers' with numbers from 1 to 100 (integers). I would like to map those numbers with dictionary. I ...
- 1
0
votes
2
answers
123
views
Swift: Efficiency considerations of shuffling an array of strings
If I have an array of strings ["xyz", "def", ...] in Swift how is this list stored?
My plan is to shuffle this array by transposing random pairs of elements.
Does the array contain ...
- 633
0
votes
1
answer
206
views
Can I preallocate a list with some size and then use the append method to populate it?
Often when I write some code I would have a rough idea of how long the list will be. I guess if I try really hard I can calculate the proper size of it at the beginning, but that's just a lot of ...
- 1
1
vote
0
answers
53
views
Graphing active times in R
I am working in R with a large dataset. The dataset records when bike rides start and end throughout a year with columns "started_at" and "ended_at" respectively. I want to create ...
1
vote
1
answer
139
views
Is it better to store a 2D image in a flat array or in an array of arrays?
I have two options.
receiving and processing pixel data in a flat array.
If I choose it, I have to use single pointer.
I have to use this code in every repeated pixel data process.
int getIndex(int ...
- 11
-1
votes
1
answer
253
views
How to speed up this code for Poincare Section using Numba?
I have the following Python code for constructing a Poincare Section for chaotic Pendulum which uses RK4 algorithm and np arrays:
import numpy as np
import matplotlib.pyplot as plt
from matplotlib ...
- 125
0
votes
1
answer
39
views
R: Improving processing time for a set of codes
Question:
How can I improve the processing time of the following code?
Goal description:
I have the following example dataset in which each observation contains
ids of two individuals (the primary ...
- 411
1
vote
2
answers
107
views
R: Improving efficiency of getting sum of differences in R
Purpose:
past vector is the set of o3.cpts found in the past period.
current vector is the set of o3.cpts found in the current period.
Each o3.cpt has a set of numbers that are associated with the ...
- 411
1
vote
2
answers
99
views
Computationally faster alternatives to calculate a new variable based on multiple columns from two large data frames in R
I have two datasets, df1 and df2.
(1) df1 has about 7 million rows and 57 columns. The columns of interest in df1 for this calculation are "county", "start.date", and "end....
- 13
0
votes
0
answers
59
views
How to shorten nested for loops and if conditions coming after each other in Python?
I have a Python script, that collects matching 5x5 crosswords from a 10000 words dataset (5 letter words). It is working, but it would take a year to be finished. Is there a way to make this code more ...
- 1
0
votes
4
answers
179
views
Nested Loop with Array running very slowly
I'm running a nested loop. I added an array in an attempt to speed it up.
When I have 100 rows and 41 columns of data in the "Active" sheet and 1000 rows and 41 columns of data in the "...
- 21
3
votes
1
answer
102
views
Curious case of efficiency of equation algorithm, why are more operations faster?
I lately wrote a bit that shall smooth out the extends of a 0-1 range. Out of curiousity I measured the performance of two possibilities to acchieve the same end result. What came out is that this:
_ =...
- 93
0
votes
0
answers
70
views
Efficiently loading/importing a big dataset that is in a non-standard format into R
The code used in this question comes from the script entitled "LASSO code (Version for Antony) in my GitHub Repository for this research project.
I am trying to load each of the 260,000 csv file-...
- 173
0
votes
2
answers
65
views
Slicing a huuuge 2D numpy ndarray - How to do this efficiently?
I have a numpy array of size (24, 131000). The first of these 24 columns contains an index corresponding to a number in the range [0, 25], for each of the 131000 rows. I want to slice this array to ...
- 77
0
votes
1
answer
398
views
Sparklyr spark_apply function on equal groups to run efficiently
How do I run a custom function efficiently in chunks within sparklyr environment?
I have a haversine function to calculate distance between 2 sets of Lat long within 1 data frame. As you can imagine, ...
- 537
-1
votes
1
answer
121
views
How can I get my python code to be more efficient? [closed]
I want to improve the efficiency of my Python code, while keeping the run time in the limit (3 seconds). How can I improve my code here?
This is the given problem: *Dan has a list of problems. The ...
- 9
0
votes
0
answers
23
views
Searching for the same date in another data frame and adding that associated row's value to the original data frame in R
I want to search through my importData for a Date and a Group that matches with my OGData. Once there is a match it would import a value into my OGData. I was able to do this doing a for loop but I ...
0
votes
1
answer
213
views
Does python create a new List after 'del' in used on an element of the List?
I plan to use an array as a stack for a binary tree print operation. Will it be more efficient to use Del operator to delete a node object from the list when it is printed, or sould I substitute some ...
0
votes
1
answer
856
views
K-Nearest Neighbor - how many reference points/features?
I want to use KNN to create a training model (I will use other ML models as well), but i'm just wondering...
I have around 6 features, with a total of let's say 60.000 (60 thousand) reference points (...
- 49
0
votes
1
answer
67
views
Multiple keywords return multiple seperate index arrays
I have a very large matrix (70k x 700k) with numeric values, is the column name for the matrix.
I want to be able to calculate the row average fo each keyword.
For example, below will calculate only ...
- 131
-6
votes
1
answer
159
views
How to (efficiently) get the start, end, and count of timeseries data from all SQL tables?
I have a massive amount of SQL tables (50,000+) each with 100,000+ time series data points. I'm just looking for the most efficient way to get the start, end, and count of each table.
I've tried the ...
- 39
1
vote
4
answers
162
views
Is there way to optimize the speed for changing values in a >2 million row data frame?
I have got a tibble of more than 2 million rows. One of the columns size is a value using M to represent million, k to represent thousand; it also has some <NA> values. The column type is ...
- 61
-1
votes
1
answer
145
views
find longest consequence of numbers so adjacents are factors or multiples
There is a puzzle to find the longest chain of numbers from 1 to 100 so that each next number should be multiple or factor for previous one. Each number can be taken once. I'm bad at programming but ...
- 1,916
0
votes
0
answers
143
views
The fastest and most efficient way to get repeated numbers in a nested array [duplicate]
I know how to do this combining a lot of if...else statements but I need a faster and more efficient way.
I need a function that will run through a nested array and return the numbers that occur more ...
- 55
0
votes
1
answer
2k
views
Google Script - Delete a row based on a blank value in a column
newish to Google Apps Script and typically tend to fall through it when I'm writing a script.
I have written the below script but it is aggressively inefficient.
The app is run against over 2k rows of ...
1
vote
0
answers
86
views
Are MS Excel's built-in conditional functions computationally more efficient than array calculations using standard operators?
The question essentially is whether built-in conditional functions in Excel such as SUMIF(), COUNTIF(), AVERAGEIF() but also SUBTOTAL() and AGGREGATE() are computationally more efficient, i.e. lighter,...
- 823
0
votes
1
answer
312
views
Getting a long list of specific elements from a list of indexes and column coordinates in Pandas
data=pd.DataFrame([[1,2,3],[21,23,24],[31,32,33]])
i=[0,1,2] # this is same as the index
y=[1,2,0]
data.iloc[x,y] gives me a 3x3 df, which I do not need.
I need to run this on a large df and would ...
-2
votes
1
answer
38
views
Most efficient way of storing and retrieving 3 lists of related data in python
I have a django model name, the model and the record of the model.
What is the most efficient way of storing and retrieving them.
I was thinking of 3 separate lists or a dictionary with the model name ...
- 31
2
votes
1
answer
44
views
How to speed up row-specific operation based on values of other variables
Say I have this data:
sysuse auto2, clear
keep if _n<=4
describe
local N = r(N)
gen a1 = price
gen a2 = mpg
gen a3 = headroom
gen a4 = trunk
gen a5 = weight
gen a6 = length
input yearA yearB
1 ...
- 2,580
1
vote
3
answers
665
views
Python/Pandas: use one column's value to be the suffix of the column name from which I want a value
I have a pandas dataframe. From multiple columns therein, I need to select the value from only one into a single new column, according to the ID (bar in this example) of that row.
I need the fastest ...
- 137