v2 Export Migration

This guide documents the SDK migration to the v2 backend's bulk-fetch endpoints for the export pipeline. The migration was shipped as a single cut-over PR (feat/SYN-6919-sdk-v2-export-migration) on top of the Phase 1 wiring already on main (RuntimeContext.v2_client + failure_classifier v2 error code support).

The change is additive for plugin authors — the legacy v1 surface is unchanged. As of SYN-7415 the v1/v2 selection switch is gone: migrated callers are now fixed to one client (see the note below) instead of choosing at runtime.

SYN-7415 — selection switch removed, callers fixed

SYN-6919 originally routed callers to v2 behind a runtime use_v2 switch, later narrowed per-category under SYN-7082. SYN-7415 removed the selection-switch machinery entirely — use_v2 / _V2_CATEGORY_POLICY / SYNAPSE_FORCE_V1_EXPORT no longer exist. There is no runtime v1/v2 branching anymore; each migrated callsite is fixed to one client:

• export/handlers.py → v2-fixed: always runs the v2 2-phase path (list → bulk_fetch/bulk_data) on ctx.v2_client. No v1 fallback — a missing v2_client raises an explicit ClientError (EXPORT_FAILED_V2_NOT_WIRED).
• to_task/steps/fetch_tasks.py → v1-fixed (legacy list_tasks).
• dataset/action.py _download_split → v1-fixed (legacy list_ground_truth_events).

The transport-agnostic 2-phase fetch helpers (_collect_then_bulk / _stream_then_bulk / strip_paginator_reserved_keys) moved from _v2_switch to synapse_sdk/plugins/actions/_v2_fetch.py. The old _v2_switch module remains only as a backward-compatible re-export shim so existing import paths keep working.

TL;DR

• 4 new v2 backend endpoints surfaced on the SDK client:
  • client.v2_client.tasks.bulk_fetch(ids) → POST /v2/tasks/bulk-fetch/
  • client.v2_client.assignments.bulk_fetch(ids) → POST /v2/assignments/bulk-fetch/
  • client.v2_client.ground_truths.bulk_data(ids) → POST /v2/ground-truths/bulk-data/
  • client.v2_client.ground_truth_events.list(...) + bulk_data(pgh_ids)
• 3 caller migrations, now fixed (SYN-7415 — no selection switch):
  • export/handlers.py (Task / Assignment / GroundTruth handlers) → v2-fixed
  • to_task/steps/fetch_tasks.py → v1-fixed
  • dataset/action.py (_download_split) → v1-fixed
• 1 nested accessor:
  • client.v2_client.ground_truth_datasets.versions(ds).events(ver).list(...)

v1 → v2 caller pattern

The v2 endpoints split the legacy single-call fetch into a 2-phase pattern:

Phase	v1 (single call)	v2 (2-phase)
List	client.list_tasks(params, list_all=True) returns heavy rows	client.v2_client.tasks.list(..., list_all=True) returns slim rows
Hydrate	(included above)	client.v2_client.tasks.bulk_fetch(ids) per chunk

The shared helper _collect_then_bulk codifies this:

from synapse_sdk.plugins.actions._v2_fetch import _collect_then_bulk

rows = _collect_then_bulk(
    list_method=lambda: client.v2_client.tasks.list(project=42, list_all=True),
    bulk_method=lambda ids: client.v2_client.tasks.bulk_fetch(ids),
    ids_per_batch=200,
)

The default extractor is compatible with both dict rows and v2 Pydantic list models. In other words, task and assignment list rows may surface as {"id": ...} or as objects such as TaskV2List / AssignmentV2List with an .id attribute, and the helper treats both shapes the same before calling bulk_fetch.

The backend caps each bulk_fetch / bulk_data call at 200 ids (V2BulkFetchMixin.BULK_FETCH_MAX_IDS in apps/shared/api/viewsets/bulk_fetch.py). The helper default matches this cap; sending more ids per call surfaces as a backend 400 ids_limit_exceeded. Callers may pass a smaller ids_per_batch to reduce per-call backpressure but never larger.

For ground_truth_events the primary key is pgh_id rather than id, so pass an extract_id callable. Keep custom extractors dict/model-compatible too:

def item_field(item, field):
    if isinstance(item, dict):
        return item[field]
    return getattr(item, field)


rows = _collect_then_bulk(
    list_method=lambda: client.v2_client.ground_truth_events.list(
        ground_truth_dataset_version=99,
        list_all=True,
    ),
    bulk_method=lambda pgh_ids: client.v2_client.ground_truth_events.bulk_data(pgh_ids),
    extract_id=lambda item: item_field(item, 'pgh_id'),
)

Throttle / retry contract on the v2 path

The v1 export handlers tuned EXPORT_THROTTLE_SECONDS / EXPORT_PAGE_RETRIES / EXPORT_READ_TIMEOUT through _apply_export_params so the v1 cursor paginator paced its own page fetches. The v2 path uses cursor pagination on the list endpoint (handled by the resource layer) plus the new bulk-fetch sequence, so _apply_export_params no longer applies wholesale.

The migrated handlers preserve the v1 inter-call throttle on the bulk-fetch sequence by forwarding EXPORT_THROTTLE_SECONDS to _collect_then_bulk(..., throttle_seconds=...). That keeps the per-handler back-pressure characteristics observed in the v1 production exporter (Task = 0.2s, GroundTruth = 0.1s, Assignment = 0.1s — all sleeps are between successive bulk calls, never before the first or after the last).

EXPORT_READ_TIMEOUT is not plumbed through to the v2 client — the v2 transport (BackendClientV2._request) applies its own timeout policy based on the v2 cursor paginator and HTTPX defaults. If you need to override v2 read-timeout semantics, set it at the BackendClientV2 construction site rather than on the handler.

Update — page-level retry is now plumbed

The export handlers now forward page-level retry to the list paginator via the pass-through (_paginator_pass_through()): page_retries=3 and retry_on_status=(502, 503, 504, 500) (EXPORT_LIST_RETRY_ON_STATUS — note the added 500 for transient DB-connection-drop responses observed on slow large-project list queries). The global DEFAULT_RETRY_ON_STATUS is unchanged, so non-export v2 consumers keep the conservative 502/503/504 default. This supersedes the "EXPORT_PAGE_RETRIES not plumbed" note above.

Large-volume resilience (keyset / presigned / retry)

The v2 export 2-phase path is hardened for large projects as follows. See Export Actions ("대량 데이터 무중단 export" section) for the full methodology and the future-work TODO list. Summary:

• Keyset list ordering — Task / Assignment list calls add sort='-id' (unique PK keyset) to eliminate DRF CursorPagination offset degradation on the non-unique -created default. GroundTruth already uses the -pgh_id event-PK keyset. Backed by a (project, id) index in synapse-backend.
• presigned bulk — bulk calls use response_mode='presigned'; _resolve_data_payload() downloads externalized data from the presigned URL via download_json instead of streaming it through the API tier.
• bounded retry — see the page-level retry info box above.

Next evolution — server-side async-job delegation (SYN-7104 P2)

The v2 2-phase path still has the SDK (ray job) pull every row over v2 REST and transform it in-process. For very large exports (100k+ rows with bulky originals) that pull is itself the cost/fragility hot-spot. SYN-7104 P2 layers backend async-job delegation on top of the v2 export path: when the target count exceeds a threshold, the data acquisition is handed to a backend async-job (server-side keyset scan → raw bundle) while the SDK enqueues, streams progress, reads the bundle, and then runs the same shared transform tail as the non-delegated path. Delegation only changes how the data is acquired — the deliverable is identical (transparent delegation, FR-8 parity). This is the first landing of the "(A안) server-side async export-to-storage" future-work item tracked in Export Actions.

This delegation is an additional layer above the 2-phase v2 path — it does not replace it. Exports below the threshold acquire data through the in-process head (v2 list → bulk-fetch); over-threshold exports acquire it through the delegated head. Either way the same shared tail (ConvertData → SaveFiles → Finalize) runs afterwards, so the plugin's custom logic applies on both paths.

Default ON · transparent delegation (staging verification recommended)

The delegation gate is enabled by default. Delegation only moves the data-acquisition path server-side; the deliverable is the same plugin format as the non-delegated path, because the delegated path runs the same shared transform tail (ConvertData → SaveFiles). The backend-written raw bundle (JSONL + files + manifest) is the delegated head's intermediate artifact, which the SDK reads (BundleRead) and feeds into the tail. To revert to in-process acquisition immediately, use the kill-switch below. Staging verification of the new delegated path is recommended before production rollout.

Gate, threshold, kill-switch

setup_steps() does not branch on the delegation decision — it always registers a single fixed 9-step registry (see Fixed step registry with shared tail below). The whole gate fires only when all of: delegation enabled (kill-switch off) AND ctx.v2_client wired AND count > threshold (strict >) AND a project_id is resolved. The decision is evaluated once (_resolve_delegation_decision()) and seeded onto context.delegation_decision; each head step's can_skip consults it so the inactive head completes-pass. The count comes from a v1 single-page count probe (no full scan); if the probe can't determine a count, the export falls back to the non-delegated (in-process head) path.

Concern	env	config	Default
Size threshold	SYNAPSE_EXPORT_ASYNC_JOB_SIZE_THRESHOLD	export_async_job_size_threshold	1000
Kill-switch (force in-process)	SYNAPSE_EXPORT_DISABLE_ASYNC_DELEGATE=1	export_disable_async_delegate	off (delegation on)

This delegation kill-switch operates only on the delegation layer — it does not change the v1/v2 selection. Export is v2-fixed (SYN-7415), so disabling delegation keeps the v2 2-phase in-process path while bypassing the server-side async-job handoff:

SYNAPSE_EXPORT_DISABLE_ASYNC_DELEGATE	Export path
off (default)	v2 2-phase, delegated when count > threshold
on	v2 2-phase in-process (delegation disabled)

# Disable only the delegation layer (keep v2 2-phase in-process):
export SYNAPSE_EXPORT_DISABLE_ASYNC_DELEGATE=1

Fixed step registry with shared tail

setup_steps() does not branch — it always registers the same fixed 9-step registry (two acquisition heads + one shared transform tail) rather than a delegation-dependent step set. The delegation decision (probed once) only flips each head step's can_skip, so the inactive head completes-pass (skip → 100%) while the shared tail runs on both paths — producing the same plugin-format deliverable (FR-8 parity):

Area	Steps	Active when delegated	Active when not
In-process head	Initialize → FetchResults → PrepareExport (v2 list → bulk-fetch)	skip (pass)	run
Delegated head	DelegateEnqueue (plugin_exports.create) → DelegateStreamProgress (async_jobs.stream_progress SSE) → BundleRead (read raw bundle + url→bundle_path rewrite)	run	skip (pass)
Shared tail	ConvertData (plugin transform) → SaveFiles → Finalize	run	run

The raw bundle (JSONL + files + manifest) the backend writes is the delegated head's intermediate artifact: BundleRead reads {prefix}/{tasks|assignments|ground_truths}.jsonl (each line {id, data, files_manifest:{spec:{file_name_original, url, bundle_path}}}), rewrites url → {prefix}/{bundle_path} via map_files_to_bundle_local (traversal-guarded, output filename kept as file_name_original), and feeds the items into the shared tail. So the delegated export still produces the plugin format — not a raw bundle.

The delegated head uses three BackendClientV2 resources — v2_client.plugin_exports.create(...), v2_client.async_jobs.stream_progress(job_id) (SSE), and v2_client.async_jobs.retrieve(job_id). The backend streams only progress (processed/total); the SDK owns event/metric/user-facing logs (set_progress step server_export, set_metrics, export_info/export_completed/export_failed, EXPORT_DELEGATED_TO_SERVER/EXPORT_DELEGATED_COMPLETED). See Export Actions — 대량 export 의 서버사이드 위임 and Steps Workflow — fixed step registry for the full contract.

Nested accessor

The events endpoint is available both as a flat resource and as a nested accessor under the owning dataset version. Both surfaces hit the same backend filter contract:

# Flat
client.v2_client.ground_truth_events.list(ground_truth_dataset_version=99)

# Nested (RPC-faithful URL)
client.v2_client.ground_truth_datasets.versions(7).events(99).list()

Use the nested form when the owning dataset / version is already part of the calling context; use the flat form for cross-dataset analytics.

Caller routing — v1/v2 is fixed, not selected

SYN-7415 removed the selection switch. Each migrated callsite is fixed to one client — there is no use_v2 call, no _V2_CATEGORY_POLICY table, and no runtime branching at the callsite:

Caller	Route	Why
export/handlers.py	v2-fixed	Export always runs the v2 2-phase path on ctx.v2_client. A missing v2_client raises an explicit ClientError (no v1 fallback).
to_task/steps/fetch_tasks.py	v1-fixed	Always the legacy client.list_tasks path.
dataset/action.py _download_split	v1-fixed	Always the legacy list_ground_truth_events path.

There is no env kill-switch to roll export back to v1 — SYNAPSE_FORCE_V1_EXPORT no longer exists. Because export is v2-fixed, an unconfigured run (no wired v2_client) fails loudly with EXPORT_FAILED_V2_NOT_WIRED instead of silently falling back to v1. If you need v1 behaviour, pin a plugin SDK version from before SYN-7415 rather than switching at runtime.

The only remaining export-related env switch is SYNAPSE_EXPORT_DISABLE_ASYNC_DELEGATE, which disables the backend delegation layer only (see Gate, threshold, kill-switch) — it does not change the v1/v2 selection.

Error envelope

The v2 backend wraps every failure in {"error": {"code": "...", "detail": "..."}}. The SDK's classify_export_error consults this envelope before the legacy v1 regex fallthrough, so callers see specific ExportLogMessageCode values (and retry policies) for known v2 codes:

error.code	ExportLogMessageCode
invalid_ids	EXPORT_FAILED_BAD_REQUEST
partial_failure	EXPORT_FAILED_BATCH_PARTIAL
rate_limited	EXPORT_FAILED_RATE_LIMITED

Unknown v2 codes fall through to v1 regex patterns; v1 regression behaviour is preserved.

Partial-failure semantics

The backend V2BulkFetchMixin silently drops unresolvable ids and returns a 200 OK with the resolved subset (specs Q2 resolution). The SDK returns whatever the backend returns — no client-side error, no exception. Callers needing strict resolution should compare len(returned) to len(requested):

ids = [1, 2, 999_999]   # 999_999 doesn't exist
rows = client.v2_client.tasks.bulk_fetch(ids)
# len(rows) == 2 — backend silently dropped the missing id.
missing = set(ids) - {row['id'] for row in rows}

files_manifest nullability

The v2 task / assignment payload exposes files_manifest as nullable per backend PR-A. Plugin code that expects file enumeration must guard against None:

def export_files(row):
    manifest = row.get('files_manifest') or {}
    for key, info in manifest.items():
        ...

GroundTruth payloads always carry a non-null files_manifest dict (backend PR-C contract), so no guard is needed for the ground_truths.bulk_data path.

Plugin author migration cheat sheet

Old (v1):

def get_results(self, client, filters):
    return client.list_tasks(params=filters, list_all=True)

New (v2-fixed export handler pattern — SYN-7415):

from synapse_sdk.plugins.actions._v2_fetch import _collect_then_bulk

def get_results(self, client, filters, *, ctx=None):
    # Export is v2-fixed: it always runs the v2 2-phase path. A missing
    # ``ctx.v2_client`` raises an explicit error (no v1 fallback).
    v2 = ctx.v2_client  # raising here if None is the handler's job
    rows = _collect_then_bulk(
        list_method=lambda: v2.tasks.list(list_all=True, **filters),
        bulk_method=lambda ids: v2.tasks.bulk_fetch(ids),
    )
    return rows, len(rows)

For a v1-fixed caller (e.g. to_task fetch or dataset download) simply keep the legacy v1 call:

def get_results(self, client, filters):
    return client.list_tasks(params=filters, list_all=True)

There is no use_v2 call in either case — each callsite is fixed to one client.

Observability

Every v2 request exposes its meta.request_id via client.v2_client.last_request_id. Use capture_request_ids() to scope the slot to a block:

with client.v2_client.capture_request_ids():
    rows = client.v2_client.tasks.bulk_fetch([1, 2, 3])
    print(client.v2_client.last_request_id)

The request_id is automatically threaded through Sentry breadcrumbs when the SDK's Sentry integration is enabled.

BC policy

• SYN-7415 fixed routing: export/handlers.py is v2-fixed — it always runs the v2 2-phase path on ctx.v2_client. A missing v2_client now raises an explicit ClientError (EXPORT_FAILED_V2_NOT_WIRED) instead of the old silent v1 fallback.
• to_task/steps/fetch_tasks.py and dataset/action.py _download_split are v1-fixed.
• The selection switch (use_v2 / _V2_CATEGORY_POLICY / SYNAPSE_FORCE_V1_EXPORT) is removed — there is no runtime v1/v2 rollback.
• The fetch helpers moved from _v2_switch to synapse_sdk/plugins/actions/_v2_fetch.py; _v2_switch remains as a backward-compatible re-export shim.
• The handler get_results(client, filters, *, ctx=None) signature keeps its kw-only ctx param.
• _collect_then_bulk accepts both dict-shaped slim rows and Pydantic list models for the default id extraction path.
• All new v2 resource methods are additive — no v1 method was removed or repurposed.

Plugins built against the existing surface will continue to work without modification. Authors migrating an export caller onto the v2 path can update their callers using the cheat sheet above.