737 questions
4
votes
2
answers
112
views
get last modified year from a file in an sftp server
I am downloading a file weekly from an sftp server.
The filenames are random so the only way to identify the new file is to use the system metadata to find the last modifed time of the files.
I can ...
0
votes
1
answer
187
views
How do I list the downloadable files using R for this NOAA website data source?
I want to download the tar.gz files from this NOAA (NCEI) data source:
https://www.ncei.noaa.gov/oa/local-climatological-data/index.html#v2/archive/
Normally, I can do something like this to get the ...
2
votes
2
answers
292
views
"Timeout was reached", "Operation too slow" with `httr2` even after setting high `httr2::req_timeout()`
I'm trying to perform a local request with httr2. I don't seem to be able to keep the request waiting for more than 10 minutes. I appreciate this is a very long time, but in my case, it is ...
0
votes
1
answer
122
views
R Install Curl tarball with RTools
I'm working on getting a server ready that for security reasons does not have access to the internet. I download files on a local computer and transfer them in.
Using miniCRAN I made a repo and ...
1
vote
0
answers
187
views
Installing RCurl fails with error "cannot find -llzma: No such file or directory"
I'm trying to install RCurl on an R 4.4.2 instance that runs on Ubuntu 22.04 (as a Jupyter Lab Docker container, spawned from a Jupyter Hub Docker container, all run on a host that is also Ubuntu 22....
0
votes
1
answer
135
views
Accessing Azure Blob using sftp protocol using R
I need to download files from an Azure blob storage with sftp support enabled. I have been provided with a username, password and endpoint which works when using filezilla to access the blob.
The ...
0
votes
1
answer
184
views
Failed to open file .curltmp. error using curl_download in Rstudio
I am trying to download an ftp file from ncbi on R-4.3.3 (Windows) but am unsuccessful as I keep getting this error below. From what I understand download.files does not work to ftp files anymore as ...
0
votes
0
answers
64
views
I am facing error when tring to download ftp link using url
this the error I am facing trying URL 'ftp://gdo-dcp.ucllnl.org/pub/dcp/subset/202404050438gr5H_n_1msibTNA'
Error in download.file(paste(url, filename, sep = ""), paste(getwd(), :
cannot ...
0
votes
1
answer
216
views
Download.file URL of zip file changed from 'http:' to 'https"
I am a little rusty as I don't use R as much as before.
I have R code that I wrote years ago that suddenly stopped working when the source URL changed from a 'http:" to "https:"
...
0
votes
1
answer
173
views
How do I use Zyte with HTTR in R
I need to use a rotating proxy IP service and opted to go with Zyte as that's what we used in my former company. I'm having trouble using the Zyte API with R. I've been messing with it for three or ...
0
votes
1
answer
769
views
R+Rcurl: How to Download a File from Pcloud?
I am trying to download a file from the cloud storage provider pcloud using R+Rcurl.
The file can be downloaded by following the link
https://e.pcloud.link/publink/show?code=...
0
votes
0
answers
152
views
RCurl cannot get files from Implicit FTP over TLS server - Access denied error 530
I'm in R trying to pull csv data from an FTP server which "Requires implicit FTP over TLS". With some help from this answer I was able to connect to the server using the following code:
# ...
0
votes
1
answer
306
views
RCurl Failed to connect to port 21 FTP Timed out
I'm in R trying to pull csv data from multiple FTP servers. So far, my code works fine in two of the servers and I'm able to gather the data I need, but a third one throws the folling error:
Error in ...
0
votes
1
answer
67
views
Find each gameId by looping through a very large list of URLs and keeping those that exist
I am trying to obtain a list of all gameId's for each boxscore url from here:
https://www.espn.com/nhl/boxscore/_/gameId/
Each URL ends with a specific gameID, e.g.
https://www.espn.com/nhl/boxscore/_/...
0
votes
1
answer
434
views
Extract binary attachment using httr
I have searched all over for an answer to this and I am coming up empty. I make a POST request to an API and it returns an object of Content-Type: application/xop+xml.
The response looks like this:
...
0
votes
1
answer
7k
views
Error in curl::curl_fetch_memory(url, handle = handle): Could not resolve host:
My R code (see below) generates these errors in some cases:
[1] "2023-08-12 16:47:37.463"
Error in curl::curl_fetch_memory(url, handle = handle): Could not resolve host: api.abc.com
Request ...
0
votes
1
answer
540
views
Republishing to RPubs, Error in curl::curl_fetch_memory(url, handle = handle)
AN ERROR OCCURS when I knit an Rmd into HTML and republish it to RPubs. There is no problem with the initial upload or republishing long ago Rmd.
Error in curl::curl_fetch_memory(url, handle = handle) ...
1
vote
0
answers
351
views
Error in curl::curl_fetch_memory(x$url$url, handle = x$url$handle) : HTTP/2 stream was not closed cleanly before end of the underlying stream
The problem occurs when I utilize the occ_search() and occ_download() functions from rgbif package. However, the error is not consistently reproducible, making it difficult to pinpoint the exact cause....
0
votes
0
answers
226
views
How to retrieve http file list from folder and download in R
I would like to get a list of downloadable files from an http website (data.mesonet.unl), and then download them. How do I do this? I've tried using the following code. but it just returns a bunch ...
3
votes
1
answer
192
views
httr & use_proxy vs RCurl behaviour difference
I am trying to use a proxy server using the use_proxy function in a GET call, The proxy server is setup rotate IPs with each call but IPs are sticking with GET & use_proxy while it is working fine ...
2
votes
0
answers
544
views
How can I get past this 'SSL connect error' when using jsonlite::fromJSON in R?
Issue
I get the following error whenever I run
all_companies <- jsonlite::fromJSON("https://www.sec.gov/files/company_tickers_exchange.json")
Warning: URL 'https://www.sec.gov/files/...
0
votes
1
answer
97
views
Get ASCII grid from compressed .gz file from URL in R
I am trying to download and gunzip grid files in ascii format, compressed to .gz files from an URL like this. I tried to get to the files via y <- gzon(url("name-of-url") and then gunzip(...
-1
votes
1
answer
26
views
Using RCurl with multiple quotes and parentheses
I am trying to get this API (that works in shell) to work within R:
curl -H 'Content-Type: text/json' -d '{"Symbols":["FLDB","APOE"]}' https://toppgene.cchmc.org/API/...
0
votes
1
answer
675
views
Accessing SFTP via Curl in Rstudio
I am trying to use the RCurl library to access an SFTP site to download files on a MacOS running Monterrey v12.4. As has happened to others, when RCurl calls the curl libraries SFTP is not enabled.
...
0
votes
0
answers
77
views
why don't my authorization header need Bearer?
I am using curl command to fetch the manifest file of the singularity image as below
curl -H "Authorization :"
and it is working fine for me and i am able to fetch the manifest file andi it ...
0
votes
1
answer
103
views
scraping with select/ option dropdown
List item
I am new to web scrapping and after a couple of Wikipedia pages I found this page where I wanted to extract the tables for all the portfolio managers. I am not able to use the things I found ...
0
votes
1
answer
3k
views
Error: package or namespace load failed for ‘RCurl’ in dyn.load(file, DLLpath = DLLpath, ...):
Error: package or namespace load failed for ‘RCurl’ in dyn.load(file, DLLpath = DLLpath, ...):
unable to load shared object '/Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/library/...
0
votes
1
answer
21
views
Using R to access Pheedloop API
My organization uses Pheedloop and I'm trying to build a dynamic solution for access its data.
So, how do I access the Pheedloop API using R? Specifically, how do I accurately submit my API ...
0
votes
1
answer
329
views
Soap request in R: POST not supported
I'd like to get a SOAP response from https://ec.europa.eu/taxation_customs/vies/checkVatTestService.wsdl, using the below example XML request:
library(RCurl)
xml.request = r'[<?xml version="1....
1
vote
1
answer
707
views
How to get response headers with `download.file`?
I have R 4.20+, so I believe utils::download.file is using capability libcurl.
I can get the headers of a url with base::curlGetHeaders(url).
Is there a parameter I can pass in download.file to return ...
2
votes
0
answers
173
views
R H2o Deep learning CURL error: Timeout was reached
I am running a H2o deep learning model in R (binary classification) using the following grid search code
hyper_params <- list(
activation = c("Rectifier", "Maxout", "...
2
votes
2
answers
503
views
Checking if URLs "exist" in R
I am trying to check if a large list of URLs "exist" in R. Let me know if you can help!
My objective: I am trying to check whether URLs from the Psychology Today online therapist directory ...
4
votes
1
answer
816
views
R - adding P12 certificate to get call
I have a successful call in Postman where I access an API with a .p12 certificate. In Postman, the certificate is stored in the global settings.
Now I am trying to translate this Postman-call to R-...
0
votes
0
answers
117
views
Reading a parquet datei from a sftp server to R
I am trying to read a Parquet file from an sftp server. Or rather I'm trying to read a lot of them at different times from different folders, so I want to automate it^^.
After a lot of googling (and ...
0
votes
0
answers
185
views
Can't install RCurl from source: dylib libomp.dynlib was built for a newer MacOS Version
I'm trying to install RCurl from source after making sure the path to curl is using the homebrew version that has sftp protocol, as the default MacOS (12.3.1) does not include that protocol:
> Sys....
5
votes
1
answer
2k
views
Faster way to download multiple files in R
I write a small downloader in R, in order to download some log files from remote server in one run:
file_remote <- fun_to_list_URLs()
file_local <- fun_to_gen_local_paths()
credentials <- &...
1
vote
1
answer
719
views
RCurl::getURL how to only list URLs of files inside a folder
I want to list the files in a remote SFTP server, so I did this:
url <- "sftp://remoteserver.com/dir/"
credentials <- "myusrname/mypwd"
file_list <- tryCatch({...
0
votes
1
answer
300
views
Putting a curl command into a R script and capturing the output to a variable
I would need to use the following bash command (or equivalent) into a R script. ideally, I would need to capture the output into a variable
curl -LH "Accept: text/bibliography; style=bibtex" ...
1
vote
1
answer
521
views
Download all files in all folders from URL
I'd like to recursively download all files from nested folders from this URL to my computer in the same nested structure:
https://hazardsdata.geoplatform.gov/?prefix=Region8/R8_MIT/Risk_MAP/Data/BLE/...
0
votes
1
answer
131
views
trying to call the package RoAuth but getting an error no package called RCurl?
: library(ROAuth) after installing but receiving Error: package or namespace load failed for ‘ROAuth’ in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]): there is no package called ‘...
1
vote
1
answer
1k
views
RCurl::ftpUpload() unspecified error: "Error in function (type, msg, asError = TRUE) : "
I am getting an odd, empty error when trying to use RCurl::ftpUpload():
RCurl::ftpUpload(what = "test.txt",
asText = FALSE,
to = "sftp://myserver.com/...
0
votes
1
answer
589
views
Download all CSV files from folders and subfolders with URL in R
I was trying to download the CSV files from this site: http://tiservice.hii.or.th/opendata/data_catalog/daily_rain/. These are the daily rainfall data, which were stored by year (in its own folder), ...
0
votes
1
answer
230
views
Validate data from website before downloading in R
I have a bunch of weather data files I want to download, but there's a mix of website url's that have data and those that don't. I'm using the download.file function in R to download the text files, ...
1
vote
1
answer
93
views
Unable to access directory of HTML site using R (RCurl package)
I am trying to access the following http directory of weather data using the RCurl package in R:
http://ncei.noaa.gov/data/global-summary-of-the-day/access/
within each directory for each year is a ...
1
vote
1
answer
63
views
knitcitations citet gives "Empty reply from server"
I am trying to compile an RMarkdown file that includes the knitcitations package. I have searched SO for "knitcitations" and did not find any similar problems to my problem.
My problem is ...
1
vote
1
answer
270
views
Read HTML table with rvest sometimes stuck and produce TimeOut Error
I had to read Dollar Rates table for each Bank from https://kursdollar.org, and I had to test this Snippets in several times:
library(stringr)
library(tidyverse)
library(rvest)
library(httr)
library(...
1
vote
2
answers
620
views
Download & decompress JSON file in R using curl or RCurl
I have the following bash script to download & decompress a JSON file:
#!/bin/sh -ex
# Ensure data directory (or a link) exists.
test -e results || mkdir results
# Download and decompress data.
...
1
vote
1
answer
68
views
RCurl and non-Roman UTF-8 characters in URL
I'm having trouble with the specific url below. The R code below generates an "invalid URL" error.
library(RCurl)
term <- "日本"
query_url <- paste("https://iss.ndl.go.jp/...
1
vote
1
answer
140
views
Loading online xml data to slider in R shiny dashboard
My friend and I built an R shiny dashboard using downloaded data. The code is as follows:
library(shiny)
library(shinydashboard)
library(dplyr)
library(tidyverse)
library(reshape)
library(scales)
ecd &...
0
votes
1
answer
140
views
Downloading CSV as plain/text from Website
I am trying to automate the download of a dataset from a website but am having trouble getting what I want. I have tried using RCurl but it is getting stuck with a tlsv1 alert protocol version error. ...