Skip to content

Commit 739fc89

Browse files
Camelia-Orcidamontenegro
Camelia-Orcid
and
amontenegro
authored
Api limiting http request filter (ORCID#7110)
* Implementation of API Rate Limiting Filter * Update development.properties * Bug fixes for slack reporting * Added the properties for test file * Added the tracking to panoply for anonymous/known users that exceeded the rate limit * changed props names * Update ApiRateLimitFilter.java to remove system.out * ApiRateLimitFilter.java log as trace when started * Update orcid-t1-web-context.xml --------- Co-authored-by: Angel Montenegro <a.montenegro@orcid.org>
1 parent 32f3b3b commit 739fc89

File tree

20 files changed

+832
-11
lines changed

20 files changed

+832
-11
lines changed

‎orcid-core/src/main/java/org/orcid/core/togglz/Features.java‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,10 @@ public enum Features implements Feature {
5353
EMAIL_DOMAINS,
5454

5555
@Label("Enable email domains in the UI")
56-
EMAIL_DOMAINS_UI;
56+
EMAIL_DOMAINS_UI,
57+
58+
@Label("Enforce rate limiting for public API when disabled the rate monitoring is on. When disabled is the mode is monitoring only.")
59+
ENABLE_PAPI_RATE_LIMITING;
5760

5861
public boolean isActive() {
5962
return FeatureContext.getFeatureManager().isActive(this);

‎orcid-core/src/main/resources/org/orcid/core/template/papi_rate_limit_email.ftl‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
<#import "email_macros.ftl" as emailMacros />
2+
Dear ${emailName},
3+
4+
This is an important message to let you know that you have exceeded our daily Public API usage limit with your integration:
5+
6+
Client Name: ${clientName}
7+
Client ID: ${clientId}
8+
9+
Please remember that the ORCID Public API is free for non-commercial use by individuals as stated in the Public APIs Terms of Service (https://info.orcid.org/public-client-terms-of-service/). By “non-commercial” we mean that you may not charge any re-use fees for the Public API, and you may not make use of the Public API in connection with any revenue-generating product or service
10+
11+
If you need access to an ORCID API for commercial use, need a higher usage quota, organizational administration of your API credentials, or the ability to write data to or access Trusted Party data in ORCID records, our Member API (https://info.orcid.org/documentation/features/member-api/) is available to ORCID member organizations.
12+
13+
To minimize any disruption to your ORCID integration in the future, we would recommend that you reach out to our Engagement Team by replying to this email to discuss our ORCID membership options.
14+
15+
Warm Regards,
16+
ORCID Support Team
17+
https://support.orcid.org
18+
<@emailMacros.msg "email.common.you_have_received_this_email" />
19+
<#include "email_footer.ftl"/>

‎orcid-core/src/main/resources/org/orcid/core/template/papi_rate_limit_email_html.ftl‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
<#import "email_macros.ftl" as emailMacros />
2+
<#escape x as x?html>
3+
<!DOCTYPE html>
4+
<html>
5+
<head>
6+
<title>${subject}</title>
7+
</head>
8+
<body>
9+
<div style="padding: 20px; padding-top: 10px; width: 700px; margin: auto;">
10+
<img src="https://orcid.org/sites/all/themes/orcid/img/orcid-logo.png" alt="ORCID.org"/>
11+
<hr />
12+
<span style="font-family: arial, helvetica, sans-serif; font-size: 15px; color: #494A4C; font-weight: bold;">Dear ${emailName},</span>
13+
<p style="font-family: arial, helvetica, sans-serif;font-size: 15px;color: #494A4C; white-space: pre;">This is an important message to let you know that you have exceeded our daily Public API usage limit with your integration:</p>
14+
<p style="font-family: arial, helvetica, sans-serif;font-size: 15px;color: #494A4C; white-space: pre;">Client Name: ${clientName}</p>
15+
<p style="font-family: arial, helvetica, sans-serif;font-size: 15px;color: #494A4C; white-space: pre;">Client ID: ${clientId}</p>
16+
<br/>
17+
<p style="font-family: arial, helvetica, sans-serif;font-size: 15px;color: #494A4C; white-space: pre;">Please remember that the ORCID Public API is free for non-commercial use by individuals as stated in the Public APIs Terms of Service (https://info.orcid.org/public-client-terms-of-service/). By “non-commercial” we mean that you may not charge any re-use fees for the Public API, and you may not make use of the Public API in connection with any revenue-generating product or service.</p>
18+
<p style="font-family: arial, helvetica, sans-serif;font-size: 15px;color: #494A4C; white-space: pre;">If you need access to an ORCID API for commercial use, need a higher usage quota, organizational administration of your API credentials, or the ability to write data to or access Trusted Party data in ORCID records, our Member API (https://info.orcid.org/documentation/features/member-api/) is available to ORCID member organizations.</p>
19+
<p style="font-family: arial, helvetica, sans-serif;font-size: 15px;color: #494A4C; white-space: pre;">To minimize any disruption to your ORCID integration in the future, we would recommend that you reach out to our Engagement Team by replying to this email to discuss our ORCID membership options.
20+
<br/>
21+
<p style="font-family: arial, helvetica, sans-serif;font-size: 15px;color: #494A4C; white-space: pre;">
22+
Warm Regards,
23+
ORCID Support Team
24+
https://support.orcid.org
25+
</p>
26+
27+
<p style="font-family: arial, helvetica, sans-serif;font-size: 15px;color: #494A4C;">
28+
<@emailMacros.msg "email.common.you_have_received_this_email" />
29+
</p>
30+
<p style="font-family: arial, helvetica, sans-serif;font-size: 15px;color: #494A4C;">
31+
<#include "email_footer_html.ftl"/>
32+
</p>
33+
</div>
34+
</body>
35+
</html>
36+
</#escape>

‎orcid-persistence/src/main/java/org/orcid/persistence/dao/PublicApiDailyRateLimitDao.java‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
package org.orcid.persistence.dao;
2+
3+
import java.time.LocalDate;
4+
5+
import org.orcid.persistence.jpa.entities.PublicApiDailyRateLimitEntity;
6+
7+
public interface PublicApiDailyRateLimitDao extends GenericDao<PublicApiDailyRateLimitEntity, Long> {
8+
9+
PublicApiDailyRateLimitEntity findByClientIdAndRequestDate(String clientId, LocalDate requestDate);
10+
PublicApiDailyRateLimitEntity findByIpAddressAndRequestDate(String ipAddress, LocalDate requestDate);
11+
int countClientRequestsWithLimitExceeded(LocalDate requestDate, int limit);
12+
int countAnonymousRequestsWithLimitExceeded(LocalDate requestDate, int limit);
13+
boolean updatePublicApiDailyRateLimit(PublicApiDailyRateLimitEntity papiRateLimitingEntity, boolean isClient);
14+
15+
}

‎orcid-persistence/src/main/java/org/orcid/persistence/dao/impl/PublicApiDailyRateLimitDaoImpl.java‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
package org.orcid.persistence.dao.impl;
2+
3+
import java.time.LocalDate;
4+
import java.time.LocalDateTime;
5+
import java.util.HashMap;
6+
import java.util.List;
7+
import java.util.Map;
8+
9+
import javax.persistence.Query;
10+
11+
import org.orcid.persistence.dao.PublicApiDailyRateLimitDao;
12+
import org.orcid.persistence.jpa.entities.PublicApiDailyRateLimitEntity;
13+
import org.slf4j.Logger;
14+
import org.slf4j.LoggerFactory;
15+
import org.springframework.transaction.annotation.Transactional;
16+
17+
public class PublicApiDailyRateLimitDaoImpl extends GenericDaoImpl<PublicApiDailyRateLimitEntity, Long> implements PublicApiDailyRateLimitDao {
18+
private static final Logger LOG = LoggerFactory.getLogger(PublicApiDailyRateLimitDaoImpl.class);
19+
20+
public PublicApiDailyRateLimitDaoImpl() {
21+
super(PublicApiDailyRateLimitEntity.class);
22+
}
23+
24+
@Override
25+
public PublicApiDailyRateLimitEntity findByClientIdAndRequestDate(String clientId, LocalDate requestDate) {
26+
Query nativeQuery = entityManager.createNativeQuery("SELECT * FROM public_api_daily_rate_limit p client_id=:clientId and requestDate=:requestDate",
27+
PublicApiDailyRateLimitEntity.class);
28+
nativeQuery.setParameter("clientId", clientId);
29+
nativeQuery.setParameter("requestDate", requestDate);
30+
List<PublicApiDailyRateLimitEntity> papiRateList = (List<PublicApiDailyRateLimitEntity>) nativeQuery.getResultList();
31+
if (papiRateList != null && papiRateList.size() > 0) {
32+
if (papiRateList.size() > 1) {
33+
LOG.warn("Found more than one entry for the daily papi rate limiting the client: " + clientId + " and request date: " + requestDate.toString());
34+
}
35+
return (PublicApiDailyRateLimitEntity) papiRateList.get(0);
36+
}
37+
return null;
38+
}
39+
40+
@Override
41+
public PublicApiDailyRateLimitEntity findByIpAddressAndRequestDate(String ipAddress, LocalDate requestDate) {
42+
String baseQuery = "SELECT * FROM public_api_daily_rate_limit p where p.ip_address=:ipAddress and p.request_date=:requestDate";
43+
44+
Query nativeQuery = entityManager.createNativeQuery(baseQuery, PublicApiDailyRateLimitEntity.class);
45+
nativeQuery.setParameter("ipAddress", ipAddress);
46+
nativeQuery.setParameter("requestDate", requestDate);
47+
48+
List<PublicApiDailyRateLimitEntity> papiRateList = (List<PublicApiDailyRateLimitEntity>) nativeQuery.getResultList();
49+
if (papiRateList != null && papiRateList.size() > 0) {
50+
LOG.debug("found results ....");
51+
if (papiRateList.size() > 1) {
52+
LOG.warn("Found more than one entry for the daily papi rate limiting, the IP Address: " + ipAddress + " and request date: " + requestDate.toString());
53+
}
54+
return (PublicApiDailyRateLimitEntity) papiRateList.get(0);
55+
}
56+
return null;
57+
}
58+
59+
public int countClientRequestsWithLimitExceeded(LocalDate requestDate, int limit) {
60+
Query nativeQuery = entityManager.createNativeQuery(
61+
"SELECT count(*) FROM public_api_daily_rate_limit p WHERE NOT ((p.client_id = '' OR p.client_id IS NULL)) and p.request_date=:requestDate and p.request_count >=:requestCount");
62+
nativeQuery.setParameter("requestDate", requestDate);
63+
nativeQuery.setParameter("requestCount", limit);
64+
List<java.math.BigInteger> tsList = nativeQuery.getResultList();
65+
if (tsList != null && !tsList.isEmpty()) {
66+
return tsList.get(0).intValue();
67+
}
68+
return 0;
69+
70+
}
71+
72+
public int countAnonymousRequestsWithLimitExceeded(LocalDate requestDate, int limit) {
73+
Query nativeQuery = entityManager.createNativeQuery(
74+
"SELECT count(*) FROM public_api_daily_rate_limit p WHERE ((p.client_id = '' OR p.client_id IS NULL)) and p.request_date=:requestDate and p.request_count >=:requestCount");
75+
nativeQuery.setParameter("requestDate", requestDate);
76+
nativeQuery.setParameter("requestCount", limit);
77+
List<java.math.BigInteger> tsList = nativeQuery.getResultList();
78+
if (tsList != null && !tsList.isEmpty()) {
79+
return tsList.get(0).intValue();
80+
}
81+
return 0;
82+
}
83+
84+
@Override
85+
@Transactional
86+
public boolean updatePublicApiDailyRateLimit(PublicApiDailyRateLimitEntity papiRateLimitingEntity, boolean isClient) {
87+
Query query;
88+
if (isClient) {
89+
query = entityManager.createNativeQuery("update public_api_daily_rate_limit set request_count = :requestCount, last_modified = now() where "
90+
+ "client_id = :clientId and request_date =:requestDate");
91+
query.setParameter("clientId", papiRateLimitingEntity.getClientId());
92+
} else {
93+
query = entityManager.createNativeQuery("update public_api_daily_rate_limit set request_count = :requestCount, last_modified = now() where "
94+
+ "ip_address = :ipAddress and request_date =:requestDate");
95+
query.setParameter("ipAddress", papiRateLimitingEntity.getIpAddress());
96+
}
97+
query.setParameter("requestCount", papiRateLimitingEntity.getRequestCount());
98+
query.setParameter("requestDate", papiRateLimitingEntity.getRequestDate().toString());
99+
return query.executeUpdate() > 0;
100+
}
101+
102+
@Override
103+
@Transactional
104+
public void persist(PublicApiDailyRateLimitEntity papiRateLimitingEntity) {
105+
String insertQuery = "INSERT INTO public_api_daily_rate_limit " + "(id, client_id, ip_address, request_count, request_date, date_created, last_modified)"
106+
+ " VALUES ( NEXTVAL('papi_daily_limit_seq'), :clientId , :ipAddress, :requestCount," + " :requestDate, now(), now())";
107+
108+
Query query = entityManager.createNativeQuery(insertQuery);
109+
query.setParameter("clientId", papiRateLimitingEntity.getClientId());
110+
query.setParameter("ipAddress", papiRateLimitingEntity.getIpAddress());
111+
query.setParameter("requestCount", papiRateLimitingEntity.getRequestCount());
112+
query.setParameter("requestDate", papiRateLimitingEntity.getRequestDate());
113+
query.executeUpdate();
114+
return;
115+
}
116+
117+
private static String logQueryWithParams(String baseQuery, Map<String, Object> params) {
118+
for (Map.Entry<String, Object> entry : params.entrySet()) {
119+
String paramPlaceholder = ":" + entry.getKey();
120+
String paramValue = (entry.getValue() instanceof String) ? "'" + entry.getValue() + "'" : entry.getValue().toString();
121+
baseQuery = baseQuery.replace(paramPlaceholder, paramValue);
122+
}
123+
return baseQuery;
124+
}
125+
126+
}

‎orcid-persistence/src/main/java/org/orcid/persistence/jpa/entities/PublicApiDailyRateLimitEntity.java‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
package org.orcid.persistence.jpa.entities;
2+
3+
import java.io.Serializable;
4+
import java.time.LocalDate;
5+
import java.util.Collection;
6+
import java.util.Date;
7+
import java.util.HashMap;
8+
import java.util.Map;
9+
10+
import javax.persistence.Column;
11+
import javax.persistence.Entity;
12+
import javax.persistence.EntityManager;
13+
import javax.persistence.GeneratedValue;
14+
import javax.persistence.GenerationType;
15+
import javax.persistence.Id;
16+
import javax.persistence.PrePersist;
17+
import javax.persistence.PreUpdate;
18+
import javax.persistence.SequenceGenerator;
19+
import javax.persistence.Table;
20+
21+
22+
@Entity
23+
@Table(name = "public_api_daily_rate_limit")
24+
public class PublicApiDailyRateLimitEntity implements OrcidEntity<Long>{
25+
26+
private static final long serialVersionUID = 7137838021634312424L;
27+
28+
@Id
29+
@GeneratedValue(strategy = GenerationType.AUTO, generator = "papi_daily_limit_seq")
30+
@SequenceGenerator(name = "papi_daily_limit_seq", sequenceName = "papi_daily_limit_seq", allocationSize = 1)
31+
private Long id;
32+
33+
@Column(name = "client_id", nullable = true)
34+
private String clientId;
35+
36+
@Column(name = "ip_address", nullable = true)
37+
private String ipAddress;
38+
39+
@Column(name = "request_count", nullable = false)
40+
private Long requestCount;
41+
42+
@Column(name = "request_date", nullable = false)
43+
private LocalDate requestDate;
44+
45+
@Column(name = "date_created", nullable = false)
46+
private Date dateCreated;
47+
48+
@Column(name = "last_modified", nullable = false)
49+
private Date lastModified;
50+
51+
public void setId(Long id) {
52+
this.id = id;
53+
}
54+
55+
public Long getId() {
56+
return id;
57+
}
58+
59+
public String getClientId() {
60+
return clientId;
61+
}
62+
63+
public void setClientId(String clientId) {
64+
this.clientId = clientId;
65+
}
66+
67+
public String getIpAddress() {
68+
return ipAddress;
69+
}
70+
71+
public void setIpAddress(String ipAddress) {
72+
this.ipAddress = ipAddress;
73+
}
74+
75+
public Long getRequestCount() {
76+
return requestCount;
77+
}
78+
79+
public void setRequestCount(Long requestCount) {
80+
this.requestCount = requestCount;
81+
}
82+
83+
public LocalDate getRequestDate() {
84+
return requestDate;
85+
}
86+
87+
public void setRequestDate(LocalDate requestDate) {
88+
this.requestDate = requestDate;
89+
}
90+
91+
92+
public Date getDateCreated() {
93+
return dateCreated;
94+
}
95+
96+
void setDateCreated(Date date) {
97+
this.dateCreated = date;
98+
}
99+
100+
public Date getLastModified() {
101+
return lastModified;
102+
}
103+
104+
void setLastModified(Date lastModified) {
105+
this.lastModified = lastModified;
106+
}
107+
108+
@PreUpdate
109+
void preUpdate() {
110+
lastModified = new Date();
111+
}
112+
113+
@PrePersist
114+
void prePersist() {
115+
Date now = new Date();
116+
dateCreated = now;
117+
lastModified = now;
118+
}
119+
120+
public static <I extends Serializable, E extends OrcidEntity<I>> Map<I, E> mapById(Collection<E> entities) {
121+
Map<I, E> map = new HashMap<I, E>(entities.size());
122+
for (E entity : entities) {
123+
map.put(entity.getId(), entity);
124+
}
125+
return map;
126+
}
127+
128+
}
129+

‎orcid-persistence/src/main/resources/META-INF/persistence.xml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,8 @@
9696
<class>org.orcid.statistics.jpa.entities.StatisticValuesEntity</class>
9797
<class>org.orcid.statistics.jpa.entities.StatisticKeyEntity</class>
9898

99+
<!-- PAPI Rate Limitig -->
100+
<class>org.orcid.persistence.jpa.entities.PublicApiDailyRateLimitEntity</class>
99101
<exclude-unlisted-classes>true</exclude-unlisted-classes>
100102

101103
<!-- <properties> -->

‎orcid-persistence/src/main/resources/db-master.xml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -405,4 +405,5 @@
405405
<include file="/db/updates/create_dw_profile_email_domain.xml" />
406406
<include file="/db/updates/add_unique_constraint_external_id_disambiguated_org.xml" />
407407
<include file="/db/updates/add_date_verified_to_email.xml" />
408+
<include file="/db/updates/create_public_api_daily_rate_limit.xml" />
408409
</databaseChangeLog>

0 commit comments

Comments
 (0)