-
Notifications
You must be signed in to change notification settings - Fork 0
added workflow export sample #2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,3 +19,4 @@ target | |
| bin/ | ||
| core/.vscode/ | ||
| .claude/ | ||
| .vscode/ | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| package io.temporal.samples.exporthistory; | ||
|
|
||
| public final class Constants { | ||
|
|
||
| public static final String QUERY = "CloseTime<=\"2025-09-30T19:43:00.000Z\""; | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For the query, should we also consider loading the workflow status to ExecutionStatus != Running?
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @alice-yin if the query has a close time, wouldn't it already be not running? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That's true. It's just in our backend, we usually call it out more specifically |
||
| public static final String ENDPOINT = "<your-namespace>.<your-account>.tmprl.cloud:7233"; | ||
| public static final String NAMESPACE = "<your-namespace>.<your-account>"; | ||
| public static final String CLIENT_CERT_PATH = "/Users/grantsmith/temporal-certs/client.pem"; | ||
| public static final String CLIENT_KEY_PATH = "/Users/grantsmith/temporal-certs/client.key"; | ||
| public static final String FILE_PATH = | ||
| "./src/main/java/io/temporal/samples/exporthistory/workflow_history.proto"; | ||
| public static final String INPUT_FILE_NAME_FROM_CLOUD_EXPORT = | ||
| "./src/test/java/io/temporal/samples/exporthistory/example-from-cloud.proto"; | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| package io.temporal.samples.exporthistory; | ||
|
|
||
| import io.grpc.netty.shaded.io.netty.handler.ssl.SslContext; | ||
| import io.temporal.client.WorkflowClient; | ||
| import io.temporal.client.WorkflowClientOptions; | ||
| import io.temporal.common.WorkflowExecutionHistory; | ||
| import io.temporal.serviceclient.SimpleSslContextBuilder; | ||
| import io.temporal.serviceclient.WorkflowServiceStubs; | ||
| import io.temporal.serviceclient.WorkflowServiceStubsOptions; | ||
| import java.io.FileInputStream; | ||
| import java.io.FileOutputStream; | ||
| import java.io.IOException; | ||
| import java.io.InputStream; | ||
| import java.util.List; | ||
| import java.util.stream.Collectors; | ||
|
|
||
| public class ExportCloudToProto { | ||
|
|
||
| public static void main(String[] args) { | ||
|
|
||
| InputStream clientCert = null; | ||
| InputStream clientKey = null; | ||
| SslContext sslContext = null; | ||
|
|
||
| try { | ||
| clientCert = new FileInputStream(Constants.CLIENT_CERT_PATH); | ||
| clientKey = new FileInputStream(Constants.CLIENT_KEY_PATH); | ||
| sslContext = SimpleSslContextBuilder.forPKCS8(clientCert, clientKey).build(); | ||
| } catch (Exception e) { | ||
| throw new RuntimeException("Error resolving file paths for mTLS: ", e); | ||
| } | ||
| WorkflowServiceStubs service = | ||
| WorkflowServiceStubs.newServiceStubs( | ||
| WorkflowServiceStubsOptions.newBuilder() | ||
| .setSslContext(sslContext) | ||
| .setTarget(Constants.ENDPOINT) | ||
| .build()); | ||
|
|
||
| WorkflowClient client = | ||
| WorkflowClient.newInstance( | ||
| service, WorkflowClientOptions.newBuilder().setNamespace(Constants.NAMESPACE).build()); | ||
|
|
||
| List<io.temporal.api.export.v1.WorkflowExecution> allExecutions = | ||
| client | ||
| // #2 change your query | ||
| .listExecutions(Constants.QUERY) | ||
| .map( | ||
| executionMetadata -> { | ||
| var exec = executionMetadata.getExecution(); | ||
| var wfid = exec.getWorkflowId(); | ||
| var rid = exec.getRunId(); | ||
| WorkflowExecutionHistory history = client.fetchHistory(wfid, rid); | ||
| io.temporal.api.history.v1.History protoHistory = history.getHistory(); | ||
|
|
||
| return io.temporal.api.export.v1.WorkflowExecution.newBuilder() | ||
| .setHistory(protoHistory) | ||
| .build(); | ||
| }) | ||
| .collect(Collectors.toList()); | ||
|
|
||
| io.temporal.api.export.v1.WorkflowExecutions executions = | ||
| io.temporal.api.export.v1.WorkflowExecutions.newBuilder() | ||
| .addAllItems(allExecutions) | ||
| .build(); | ||
|
|
||
| byte[] binary = executions.toByteArray(); | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we serialize to JSON before writing?
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @alice-yin I think the intent of this sample is to match what the cloud export feature exports... and its export is proto, not json, right? |
||
|
|
||
| FileOutputStream fos = null; | ||
| try { | ||
|
|
||
| fos = new FileOutputStream(Constants.FILE_PATH); | ||
|
|
||
| fos.write(binary); | ||
|
|
||
| System.out.println("Workflow history saved to: " + Constants.FILE_PATH); | ||
|
|
||
| } catch (IOException e) { | ||
| System.err.println("An error occurred while writing the file: " + e.getMessage()); | ||
| e.printStackTrace(); | ||
| } finally { | ||
| try { | ||
| if (fos != null) { | ||
| fos.close(); | ||
| } | ||
| } catch (IOException e) { | ||
| System.err.println("Error closing the file stream: " + e.getMessage()); | ||
| e.printStackTrace(); | ||
| } | ||
| } | ||
|
|
||
| System.exit(0); | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| # Export History Sample | ||
|
|
||
| This sample shows how to get the execution history for some workflows in a namespace. | ||
|
|
||
| It can be used in situations in which the workflow export feature was turned on, and it | ||
| is desireable to get the history of the workflows that existed before the time it was turned | ||
| on. | ||
|
|
||
| It reads workflow executions from a cloud namespace and writes them to | ||
| `./src/main/java/io/temporal/samples/exporthistory/workflow_history.proto`. | ||
|
|
||
| ## Instructions | ||
|
|
||
| 1. In Constants.java, update the connection config (namespace, endpoint, file paths for the certs, etc), and | ||
| modify the query to suit your needs. | ||
| 2. Run the following command to run the code | ||
| `./gradlew -q execute -PmainClass=io.temporal.samples.exporthistory.ExportCloudToProto` | ||
|
|
||
| ## Caveats and Considerations | ||
|
|
||
| - Rate limits were not considered when writing this sample. | ||
| - Internal/system workflows (for example, schedules) will differ. | ||
| - If comparing to the cloud export feature, search attributes will differ. The cloud export | ||
| feature exports the internal search attribute names, but this script retrieves the user facing names. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,112 @@ | ||
| package io.temporal.samples.exporthistory; | ||
|
|
||
| import static org.junit.Assert.assertTrue; | ||
|
|
||
| import com.google.protobuf.util.Timestamps; | ||
| import io.temporal.api.export.v1.WorkflowExecution; | ||
| import io.temporal.api.export.v1.WorkflowExecutions; | ||
| import io.temporal.api.history.v1.History; | ||
| import io.temporal.api.history.v1.HistoryEvent; | ||
| import io.temporal.api.history.v1.StartChildWorkflowExecutionInitiatedEventAttributes; | ||
| import io.temporal.api.history.v1.UpsertWorkflowSearchAttributesEventAttributes; | ||
| import io.temporal.api.history.v1.WorkflowExecutionStartedEventAttributes; | ||
| import java.io.FileInputStream; | ||
| import java.io.IOException; | ||
| import java.util.ArrayList; | ||
| import org.junit.Test; | ||
|
|
||
| public class ExportHistoryTest { | ||
|
|
||
| @Test | ||
| public void testExportEquality() { | ||
|
|
||
| WorkflowExecutions executionsFromScript; | ||
| try (FileInputStream fis = new FileInputStream(Constants.FILE_PATH)) { | ||
| executionsFromScript = WorkflowExecutions.parseFrom(fis); | ||
| } catch (IOException e) { | ||
| throw new RuntimeException("Error reading proto file: ", e); | ||
| } | ||
|
|
||
| WorkflowExecutions executionsFromCloudExport; | ||
| try (FileInputStream fis = new FileInputStream(Constants.INPUT_FILE_NAME_FROM_CLOUD_EXPORT)) { | ||
| executionsFromCloudExport = WorkflowExecutions.parseFrom(fis); | ||
| } catch (IOException e) { | ||
| throw new RuntimeException("Error reading proto file: ", e); | ||
| } | ||
|
|
||
| assertTrue(equalIgnoringSearchAttributes(executionsFromScript, executionsFromCloudExport)); | ||
| } | ||
|
|
||
| /** Compare two WorkflowExecutions while ignoring all search_attributes. */ | ||
| public static boolean equalIgnoringSearchAttributes(WorkflowExecutions a, WorkflowExecutions b) { | ||
| var aItems = new ArrayList<>(a.getItemsList()); | ||
| aItems.sort( | ||
| (x, y) -> | ||
| Timestamps.compare( | ||
| x.getHistory().getEvents(0).getEventTime(), | ||
| y.getHistory().getEvents(0).getEventTime())); | ||
| var bItems = new ArrayList<>(b.getItemsList()); | ||
| bItems.sort( | ||
| (x, y) -> | ||
| Timestamps.compare( | ||
| x.getHistory().getEvents(0).getEventTime(), | ||
| y.getHistory().getEvents(0).getEventTime())); | ||
| if (aItems.size() != bItems.size()) return false; | ||
|
|
||
| for (int i = 0; i < aItems.size(); i++) { | ||
| WorkflowExecution sa = stripSearchAttrs(aItems.get(i)); | ||
| WorkflowExecution sb = stripSearchAttrs(bItems.get(i)); | ||
| if (!sa.equals(sb)) { | ||
| return false; | ||
| } | ||
| } | ||
| return true; | ||
| } | ||
|
|
||
| private static WorkflowExecution stripSearchAttrs(WorkflowExecution exec) { | ||
| if (!exec.hasHistory()) return exec; | ||
| History sanitized = stripSearchAttrs(exec.getHistory()); | ||
| return exec.toBuilder().setHistory(sanitized).build(); | ||
| } | ||
|
|
||
| /** Strip search_attributes from a History (per-event). */ | ||
| private static History stripSearchAttrs(History h) { | ||
| History.Builder hb = h.toBuilder(); | ||
| for (int i = 0; i < hb.getEventsCount(); i++) { | ||
| hb.setEvents(i, stripSearchAttrs(hb.getEvents(i))); | ||
| } | ||
| return hb.build(); | ||
| } | ||
|
|
||
| /** Strip search_attributes from any event attributes that carry them. */ | ||
| private static HistoryEvent stripSearchAttrs(HistoryEvent e) { | ||
| HistoryEvent.Builder eb = e.toBuilder(); | ||
|
|
||
| // 1) WorkflowExecutionStartedEventAttributes | ||
| if (e.hasWorkflowExecutionStartedEventAttributes()) { | ||
| WorkflowExecutionStartedEventAttributes.Builder a = | ||
| e.getWorkflowExecutionStartedEventAttributes().toBuilder(); | ||
| a.clearSearchAttributes(); | ||
| eb.setWorkflowExecutionStartedEventAttributes(a); | ||
| } | ||
|
|
||
| // 2) UpsertWorkflowSearchAttributesEventAttributes | ||
| if (e.hasUpsertWorkflowSearchAttributesEventAttributes()) { | ||
| UpsertWorkflowSearchAttributesEventAttributes.Builder up = | ||
| e.getUpsertWorkflowSearchAttributesEventAttributes().toBuilder(); | ||
| up.clearSearchAttributes(); | ||
| eb.setUpsertWorkflowSearchAttributesEventAttributes(up); | ||
| } | ||
|
|
||
| // 3) StartChildWorkflowExecutionInitiatedEventAttributes (often carries search_attributes) | ||
| if (e.hasStartChildWorkflowExecutionInitiatedEventAttributes()) { | ||
| StartChildWorkflowExecutionInitiatedEventAttributes.Builder child = | ||
| e.getStartChildWorkflowExecutionInitiatedEventAttributes().toBuilder(); | ||
| // This field is optional in some server versions; clear if present. | ||
| child.clearSearchAttributes(); | ||
| eb.setStartChildWorkflowExecutionInitiatedEventAttributes(child); | ||
| } | ||
|
|
||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think there is another case for Continue as new case.See https://github.com/temporalio/temporal/blob/main/service/history/api/get_history_util.go#L360 If this is what trying for. |
||
| return eb.build(); | ||
| } | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider this to be one of the input?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also consider the query to be startTime, closeTime.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@alice-yin what do you mean? Are you saying it should be a command line arg?
@alice-yin I thought the export was based on close time, right?
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think if we allow startTime and closeTime as both in command line arg. On a second thought, how about we just allow QUERY as the whole input? As for query, user could directly get the input from the UI. That's probably easier for user as well.