新增快手平台和对应的接口
This commit is contained in:
29
.codebuddy/memory/2026-06-01.md
Normal file
29
.codebuddy/memory/2026-06-01.md
Normal file
@@ -0,0 +1,29 @@
|
||||
# 2026-06-01 工作记录
|
||||
|
||||
## 快手代发订单详情新增
|
||||
- 新增 `open.dropshipping.order.merchant.detail` 到 `seed_data_kuaishou.sql`
|
||||
- 第19个接口，POST、prefetch 自 dropshipping_order_list、single_record
|
||||
|
||||
## 快手接口全面审查
|
||||
发现 prefetch 相关7个接口存在3个严重Bug：
|
||||
1. Prefetch 阶段业务参数未包入 param JSON（body_wrapper_field 未生效）
|
||||
2. Prefetch 阶段 `parseResp(nil config)` 无法解析 data.items/data.cpsOrderList 等路径
|
||||
3. Prefetch 循环未处理游标分页（固定页码递增）
|
||||
|
||||
## Bug 修复（dynamic_sync.go）
|
||||
1. `syncWithPrefetch` 改用 `buildReqBody(prefetchIface)` 构建请求，body_wrapper_field 正确包装
|
||||
2. 改用 `parseRespExt(resp.Body, prefetchIface.ResponseConfig)` 解析响应
|
||||
3. 支持游标分页的 prefetch 循环（cursor/pcursor）
|
||||
4. `buildPrefetchParams` 增加过滤 body_wrapper_field/exclude_from_wrapper/cursor_pagination/time_field_mode
|
||||
5. 新增 `collectPrefetchEntities` 辅助函数
|
||||
6. 修复并发阶段 `inQuery` 变量缺失的问题
|
||||
|
||||
## 同步调度并发锁修复
|
||||
1. `SyncByConfig` 新增 `syncRunningMap` 内存锁（sync.Map.LoadOrStore），防止同一接口并发执行
|
||||
2. 调度器从 `time.NewTicker` 改为 `for { run(); time.Sleep(interval) }`，前一次完成后才开始计时
|
||||
|
||||
## MEMORY.md 更新
|
||||
- response_config 增加 `single_record` 字段说明
|
||||
- prefetch 流程补充游标分页、参数构建、响应解析的详细说明
|
||||
- 新增「并发保护」章节，记录内存锁和调度器时序
|
||||
- 调度器章节补充 for+sleep 模式说明
|
||||
479
.codebuddy/memory/MEMORY.md
Normal file
479
.codebuddy/memory/MEMORY.md
Normal file
@@ -0,0 +1,479 @@
|
||||
# 数据引擎 - 项目记忆 (MEMORY.md)
|
||||
|
||||
## 项目概述
|
||||
Golang + GoFrame v2 + PostgreSQL 的配置化数据同步引擎。通过 `api_datasource_platform` + `api_interface` 两表驱动，支持多平台多接口的数据同步，无需重新编译代码。
|
||||
|
||||
## 核心表结构
|
||||
见 `sql/init_core_tables.sql`：
|
||||
- `api_datasource_platform` — 数据源平台配置
|
||||
- `api_interface` — API 接口配置（request_config / response_config / table_definition 三大 JSONB）
|
||||
- `sync_tracker` — 同步跟踪（last_sync_time, sync_status）
|
||||
- `sync_task_log` — 同步任务日志（用于补偿重试）
|
||||
|
||||
## 认证类型（auth_type）模板
|
||||
|
||||
### 1. OAuth2（腾讯广告）
|
||||
```
|
||||
auth_type: OAUTH2
|
||||
token: access_token 值
|
||||
client_id / client_secret: OAuth2 凭证
|
||||
auth_config: {
|
||||
"token_in_query": true, // token 放 URL 查询参数
|
||||
"query_key": "access_token", // 参数名
|
||||
"refresh_token": "xxx", // 刷新 token
|
||||
"extra_query_params": {
|
||||
"timestamp": "{timestamp}",
|
||||
"nonce": "{nonce}"
|
||||
}
|
||||
}
|
||||
```
|
||||
安全校验：`applyAuthHeader` 中 OAUTH2/TOKEN 模式会设 `Authorization: Bearer {token}`，但 `token_in_query=true` 时跳过 Header，转加 URL 参数。
|
||||
|
||||
### 2. API_KEY + 签名（快手电商）
|
||||
```
|
||||
auth_type: API_KEY
|
||||
token/access_token: 调用凭证
|
||||
auth_config: {
|
||||
"sign_algorithm": "md5_upper", // 签名算法: md5 / md5_upper
|
||||
"app_key": "xxx",
|
||||
"app_secret": "xxx",
|
||||
"token_in_query": true,
|
||||
"query_key": "access_token",
|
||||
"extra_query_params": {
|
||||
"timestamp": "{timestamp_ms}",
|
||||
"signMethod": "MD5"
|
||||
}
|
||||
}
|
||||
```
|
||||
签名过程：所有查询参数按 key 排序 → `k1=v1&k2=v2&...&key=app_secret` → MD5 摘要 → 追加 `sign` 参数。
|
||||
`{timestamp_ms}` 替换为毫秒时间戳，`{timestamp}` 替换为秒时间戳，`{nonce}` 替换为随机数。
|
||||
|
||||
### 3. TOKEN（简单 Bearer Token）
|
||||
```
|
||||
auth_type: TOKEN
|
||||
token: "xxx"
|
||||
```
|
||||
`applyAuthHeader` 设 `Authorization: Bearer {token}`。
|
||||
|
||||
### 4. SIGN（仅签名，无额外 token）
|
||||
```
|
||||
auth_type: SIGN
|
||||
auth_config: { "app_key": "...", "app_secret": "..." }
|
||||
```
|
||||
`applyAuthHeader` 不设特殊 Header，仅通过 `applySignature` 追加签名参数。`PlatformManager` 从 `auth_config` 读取 app_key/app_secret。
|
||||
|
||||
## request_config 字段完整参考
|
||||
|
||||
| 字段 | 类型 | 含义 | 适用平台 |
|
||||
|------|------|------|---------|
|
||||
| `parameters_location` | string | `"query"`=URL查询参数，不设则POST走JSON body | 通用 |
|
||||
| `page_param` | string | 分页参数名，默认`"page"` | 通用 |
|
||||
| `page_size_param` | string | 每页条数参数名，默认`"page_size"` | 通用 |
|
||||
| `page_size` | int | 每页条数，**覆盖 config.yml 全局默认值**，需按各平台最大限制设置（如快手最大50，腾讯100） | 通用 |
|
||||
| `cursor_pagination` | bool | 是否游标分页（true=游标，false/无=普通分页） | 通用 |
|
||||
| `time_field` | string | 增量时间字段名 | 通用 |
|
||||
| `time_field_mode` | string | `"range"`=beginTime/endTime模式（快手），`"filtering"`=filtering数组模式（腾讯，默认） | 通用 |
|
||||
| `prefetch` | object | 预取配置（见下方） | 需遍历实体的接口 |
|
||||
| `body_wrapper_field` | string | 业务参数包装字段名，如`"param"` | 快手 |
|
||||
| `exclude_from_wrapper` | string[] | body_wrapper 时不包装的字段 | 快手 |
|
||||
| `top_level_params` | string[] | 保留在顶层的字段（备用） | 通用 |
|
||||
| `fields` | string[] | API 的 fields 参数 | 腾讯 |
|
||||
|
||||
### prefetch 预取配置
|
||||
```json
|
||||
"prefetch": {
|
||||
"url": "/advertiser/get", // 预取接口的相对路径
|
||||
"method": "GET", // HTTP 方法
|
||||
"response_path": "data.list", // 从响应中取列表的 JSON 路径
|
||||
"target_param": "account_id", // 预取值注入到数据接口的参数名
|
||||
"value_field": "account_id" // 从预取列表里取哪个字段的值
|
||||
}
|
||||
```
|
||||
流程：先分页拉取预取接口 → 提取所有 ID → 逐个 ID 并发拉取数据接口 → 将 ID 注入每行数据。
|
||||
|
||||
**预取阶段分页**：自动适配预取来源接口的分页方式：
|
||||
- **游标分页**（`cursor_pagination: true`）：首次 `cursor=""`，后续从响应 `cursor_field` 取值，直到 `""` 或 `"nomore"` 停止
|
||||
- **普通分页**：读取 `page_info.total_page` 遍历全部页码
|
||||
|
||||
**预取阶段参数构建**：使用 `buildReqBody(prefetchIface)` 构建，自动处理 `body_wrapper_field` 包装（param JSON）、时间过滤（增量时传 beginTime/endTime）、分页参数名。响应解析使用 `prefetchIface.ResponseConfig`，正确识别各种 `list_path`（`data.orderList`、`data.items`、`data.cpsOrderList` 等）。
|
||||
|
||||
**预取阶段时间过滤**：增量同步时自动复用预取来源接口的 `time_field` + `time_field_mode`（如果来源支持增量，prefetch 会自动带上时间范围只拉增量数据）。
|
||||
|
||||
## response_config 字段完整参考
|
||||
|
||||
| 字段 | 类型 | 含义 |
|
||||
|------|------|------|
|
||||
| `success_field` | string | 成功标识字段名，默认`"code"` |
|
||||
| `success_value` | number | 成功标识值，默认`0` |
|
||||
| `message_field` | string | 错误信息字段名，默认`"message"` |
|
||||
| `list_path` | string | 数据列表在 JSON 中的路径，如`"data.list"`、`"data.orderList"` |
|
||||
| `cursor_field` | string | 游标字段路径，如`"data.cursor"` |
|
||||
| `cursor_end_marker` | string | 游标结束标记，如`"nomore"`（代码中硬判断此字符串） |
|
||||
| `single_record` | bool | 单记录模式：`list_path` 指向的路径是单个对象而非数组，会自动包装为单元素数组 |
|
||||
|
||||
**响应解析逻辑**：`parseRespExt` 函数
|
||||
1. 检查 `success_field` 的值是否等于 `success_value` | **注意**：`success_value` 仅支持数字类型（内部通过 `toFloat64` 转换），不支持字符串如 `"SUCCESS"` 或布尔值 `true`
|
||||
2. 按 `list_path` 遍历到数据列表数组 | 支持路径如 `data.orderList`，最后一段是数组 OK（代码有兼容处理）
|
||||
3. 每条数据调用 `flattenRow` 展平嵌套 field（`orderBaseInfo.oid` → `oid`）
|
||||
4. 提取 `cursor_field` 用于下一页，为空或等于 `cursor_end_marker` 时停止循环
|
||||
5. 从 `page_info.total_page` 获取总页数（普通分页用）
|
||||
|
||||
**游标分页首次请求**：代码自动传 `cursor=""`（首页不传游标），由服务端返回第一页数据 + 下一页游标值。
|
||||
|
||||
**单记录模式**：`"single_record": true` 时，`list_path` 指向单个对象（如 `"data"`），自动包装成单元素数组用于统一处理。
|
||||
|
||||
## table_definition 字段完整参考
|
||||
|
||||
| 字段 | 类型 | 含义 |
|
||||
|------|------|------|
|
||||
| `table_name` | string | 目标表名，如`"kuaishou_order_list"` |
|
||||
| `columns` | array | 列定义数组，每个元素 `{"name":"字段名","type":"PG类型","comment":"说明"}` |
|
||||
| `conflict_keys` | string[] | upsert 冲突键，如`["oid"]`或`["image_id","account_id"]` |
|
||||
|
||||
注意：
|
||||
- 列名必须和 API 响应中的字段名**完全一致**（camelCase/snake_case 取决于 API）
|
||||
- 嵌套字段会被 `flattenRow` 自动展平到顶层
|
||||
- `raw_data` 列自动包含，存储完整原始 JSON
|
||||
- 自动建表时会额外添加 id/tenant_id/creator/created_at/updater/updated_at/deleted_at 审计字段
|
||||
- **必须写全**：`columns` 中需列出 API 响应中所有平铺的标量字段（字符串/数字/布尔），数组和嵌套对象字段保留在 `raw_data` 中即可
|
||||
|
||||
## 同步策略原则
|
||||
- **API 支持时间过滤**（有 beginTime/endTime 或 filtering 参数）→ 配 `time_field` + `time_field_mode`，实现增量同步
|
||||
- **API 不支持时间过滤** → 不配 `time_field`，走全量同步
|
||||
- **详情/子接口**（通过 prefetch 遍历）：prefetch 阶段自动复用预取来源接口的 `time_field_mode`（如果来源接口支持增量，prefetch 也会带上时间过滤只拉增量数据）
|
||||
|
||||
## 全量同步流程
|
||||
|
||||
```
|
||||
SyncByConfig(platformCode, interfaceCode, isFullSync=false)
|
||||
│
|
||||
├─ getLastSyncTime() → 无记录返回 0
|
||||
│ lastSyncTime=0 → 全量
|
||||
│
|
||||
├─ markSyncRunning() 标记 running
|
||||
│
|
||||
├─ buildReqBody(page=1, cursor="", lastSyncTime=0)
|
||||
│ ├─ 游标分页: cursor="" 首次空游标
|
||||
│ ├─ range 模式: beginTime=90天前, endTime=now
|
||||
│ └─ filtering 模式: 不添加时间过滤
|
||||
│
|
||||
├─ API → parseRespExt → flattenRow → savePage(upsert)
|
||||
│
|
||||
└─ 游标循环 → 直到 cursor="" 或 "nomore"
|
||||
updateSyncTime() 记录最大时间戳
|
||||
```
|
||||
|
||||
## 增量同步流程
|
||||
|
||||
```
|
||||
SyncByConfig(platformCode, interfaceCode, isFullSync=false)
|
||||
│
|
||||
├─ getLastSyncTime() → 返回上次 maxTime
|
||||
│ lastSyncTime>0 → 增量
|
||||
│
|
||||
├─ buildReqBody(cursor="", lastSyncTime=xxx)
|
||||
│ ├─ range 模式: beginTime=lastSyncTime, endTime=now
|
||||
│ └─ filtering 模式: filtering=[{field=time_field, operator=GREATER_EQUALS, values=[lastSyncTime]}]
|
||||
│
|
||||
└─ 后续同全量（只拉取时间范围内的数据）
|
||||
```
|
||||
|
||||
## 并发保护
|
||||
|
||||
### 内存锁（同一进程内）
|
||||
`SyncByConfig` 入口使用 `sync.Map`（`syncRunningMap`）记录正在执行的接口。若同一接口的同步请求再次到达（如调度器重叠），直接跳过并打印警告：
|
||||
```
|
||||
WARN 接口 [kuaishou/order_list] 正在同步中，跳过重复请求
|
||||
```
|
||||
无论接口类型（prefetch/非prefetch、腾讯/快手、现有/新增）均自动生效。
|
||||
|
||||
### 异常中断检测（DB 状态）
|
||||
如果 `sync_tracker.sync_status = "running"`，说明上次同步异常中断（进程崩溃），自动回退全量。
|
||||
|
||||
## 补偿机制
|
||||
`compensation.go` 定时扫描 `sync_task_log` 中 status="failed" 的任务，自动调用 `SyncByConfig` 重试（增量），支持指数退避（5, 15, 30, 60, 120 分钟）。
|
||||
|
||||
## 自动同步调度
|
||||
`sync_scheduler.go` 的 `runAutoSync` 遍历所有 ACTIVE 平台下的所有配置了 table_definition 的活跃接口，自动调用同步。
|
||||
|
||||
**调度器时序**：使用 `for { runAutoSync(); time.Sleep(interval) }` 模式，而非 `time.NewTicker`。每次同步**完全结束后**才开始计时 interval，避免前一次未跑完就启动下一次导致重叠。无论全量跑 30 分钟还是 70 分钟，下一次都在「本次完成时间 + interval」后执行。
|
||||
|
||||
## 新增平台 + 接口的操作步骤
|
||||
|
||||
### 操作步骤
|
||||
1. 在 `sql/` 下创建 `seed_data_{平台编码}.sql`
|
||||
2. 先执行 `sql/init_core_tables.sql`（仅首次需要）
|
||||
3. 执行新建的 seed SQL 文件（INSERT 平台 → INSERT 接口）
|
||||
4. 配置认证凭据（token / app_key / app_secret 等）
|
||||
5. 重启服务，管理后台验证：`http://localhost:3002/admin`
|
||||
|
||||
### 种子 SQL 标准模板
|
||||
|
||||
#### 模板 A：OAuth2 + 普通分页（腾讯广告风格）
|
||||
```sql
|
||||
-- =============================================
|
||||
-- 1. 创建 {平台编码} 平台
|
||||
-- =============================================
|
||||
INSERT INTO api_datasource_platform (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_code, platform_name, description, status,
|
||||
api_base_url, auth_type,
|
||||
token, client_id, client_secret,
|
||||
auth_config,
|
||||
rate_limit_per_minute, rate_limit_per_hour,
|
||||
concurrency_limit, request_timeout_ms, max_retries, retry_delay_ms
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
'{平台编码}', '{平台名称}', '{描述}', 'ACTIVE',
|
||||
'{API_BASE_URL}', 'OAUTH2',
|
||||
'{access_token}', '{client_id}', '{client_secret}',
|
||||
'{
|
||||
"token_in_query": true,
|
||||
"query_key": "access_token",
|
||||
"refresh_token": "{refresh_token}",
|
||||
"extra_query_params": {
|
||||
"timestamp": "{timestamp}",
|
||||
"nonce": "{nonce}"
|
||||
}
|
||||
}'::jsonb,
|
||||
60, 3600, 10, 30000, 3, 1000
|
||||
);
|
||||
|
||||
-- 2. 创建 {接口名} 接口（普通分页 + 增量 filtering）
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = '{平台编码}'),
|
||||
'{接口名称}', '{接口编码}',
|
||||
'{相对路径}', 'GET', 'active', 'inherit',
|
||||
'{
|
||||
"parameters_location": "query",
|
||||
"page": 1,
|
||||
"page_size": 100,
|
||||
"page_param": "page",
|
||||
"page_size_param": "page_size",
|
||||
"pagination_mode": "PAGINATION_MODE_NORMAL",
|
||||
"time_field": "last_modified_time",
|
||||
"fields": ["field1", "field2"]
|
||||
}'::jsonb,
|
||||
'{
|
||||
"success_field": "code",
|
||||
"success_value": 0,
|
||||
"message_field": "message",
|
||||
"list_path": "data.list"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "{表名}",
|
||||
"columns": [
|
||||
{"name": "id", "type": "BIGINT", "comment": "主键"},
|
||||
{"name": "name", "type": "VARCHAR(200)", "comment": "名称"},
|
||||
{"name": "created_time", "type": "BIGINT", "comment": "创建时间"}
|
||||
],
|
||||
"conflict_keys": ["id"]
|
||||
}'::jsonb
|
||||
);
|
||||
```
|
||||
|
||||
#### 模板 B：OAuth2 + prefetch 预取（遍历账户拉取子数据，腾讯广告风格）
|
||||
```sql
|
||||
-- 预取接口（先拉取实体列表）
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = '{平台编码}'),
|
||||
'{实体列表名称}', '{实体编码}',
|
||||
'{实体列表相对路径}', 'GET', 'active', 'inherit',
|
||||
'{
|
||||
"parameters_location": "query",
|
||||
"page": 1,
|
||||
"page_size": 100,
|
||||
"page_param": "page",
|
||||
"page_size_param": "page_size",
|
||||
"fields": ["id", "name"]
|
||||
}'::jsonb,
|
||||
'{
|
||||
"success_field": "code",
|
||||
"success_value": 0,
|
||||
"list_path": "data.list"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "{实体表名}",
|
||||
"columns": [
|
||||
{"name": "id", "type": "BIGINT", "comment": "实体ID"},
|
||||
{"name": "name", "type": "VARCHAR(200)", "comment": "实体名称"}
|
||||
],
|
||||
"conflict_keys": ["id"]
|
||||
}'::jsonb
|
||||
);
|
||||
|
||||
-- 数据接口（遍历每个实体拉取数据）
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = '{平台编码}'),
|
||||
'{数据接口名称}', '{数据接口编码}',
|
||||
'{数据接口相对路径}', 'GET', 'active', 'inherit',
|
||||
'{
|
||||
"parameters_location": "query",
|
||||
"page": 1,
|
||||
"page_size": 100,
|
||||
"page_param": "page",
|
||||
"page_size_param": "page_size",
|
||||
"time_field": "last_modified_time",
|
||||
"prefetch": {
|
||||
"url": "{实体列表相对路径}",
|
||||
"method": "GET",
|
||||
"response_path": "data.list",
|
||||
"target_param": "{参数名}",
|
||||
"value_field": "{取值字段}"
|
||||
}
|
||||
}'::jsonb,
|
||||
'{
|
||||
"success_field": "code",
|
||||
"success_value": 0,
|
||||
"list_path": "data.list"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "{数据表名}",
|
||||
"columns": [
|
||||
{"name": "id", "type": "BIGINT", "comment": "ID"},
|
||||
{"name": "{参数名}", "type": "BIGINT", "comment": "实体ID（预取注入）"},
|
||||
{"name": "created_time", "type": "BIGINT", "comment": "创建时间"}
|
||||
],
|
||||
"conflict_keys": ["id", "{参数名}"]
|
||||
}'::jsonb
|
||||
);
|
||||
```
|
||||
|
||||
#### 模板 C：API_KEY + MD5 签名 + 游标分页 + param 包装（快手风格）
|
||||
```sql
|
||||
INSERT INTO api_datasource_platform (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_code, platform_name, description, status,
|
||||
api_base_url, auth_type,
|
||||
auth_config,
|
||||
rate_limit_per_minute, rate_limit_per_hour,
|
||||
concurrency_limit, request_timeout_ms, max_retries, retry_delay_ms
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
'{平台编码}', '{平台名称}', '{描述}', 'ACTIVE',
|
||||
'{API_BASE_URL}', 'API_KEY',
|
||||
'{
|
||||
"sign_algorithm": "md5_upper",
|
||||
"app_key": "{YOUR_APP_KEY}",
|
||||
"app_secret": "{YOUR_APP_SECRET}",
|
||||
"token_in_query": true,
|
||||
"query_key": "access_token",
|
||||
"extra_query_params": {
|
||||
"timestamp": "{timestamp_ms}",
|
||||
"signMethod": "MD5"
|
||||
}
|
||||
}'::jsonb,
|
||||
100, 3600, 5, 30000, 3, 1000
|
||||
);
|
||||
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = '{平台编码}'),
|
||||
'{接口名称}', '{接口编码}',
|
||||
'{相对路径}', 'GET', 'active', 'inherit',
|
||||
'{
|
||||
"parameters_location": "query",
|
||||
"page_param": "cursor",
|
||||
"page_size_param": "pageSize",
|
||||
"cursor_pagination": true,
|
||||
"method": "{API方法名}",
|
||||
"version": 1,
|
||||
"signMethod": "MD5",
|
||||
"pageSize": 50,
|
||||
"time_field": "updateTime",
|
||||
"time_field_mode": "range",
|
||||
"body_wrapper_field": "param",
|
||||
"exclude_from_wrapper": ["method", "version", "signMethod"],
|
||||
"{其他业务参数}": {值}
|
||||
}'::jsonb,
|
||||
'{
|
||||
"success_field": "result",
|
||||
"success_value": 1,
|
||||
"list_path": "data.{数组字段名}",
|
||||
"cursor_field": "data.cursor",
|
||||
"cursor_end_marker": "nomore"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "{表名}",
|
||||
"columns": [
|
||||
{"name": "oid", "type": "BIGINT", "comment": "订单ID"},
|
||||
{"name": "createTime", "type": "BIGINT", "comment": "创建时间"},
|
||||
{"name": "updateTime", "type": "BIGINT", "comment": "更新时间"},
|
||||
{"name": "{字段名}", "type": "{类型}", "comment": "{说明}"}
|
||||
],
|
||||
"conflict_keys": ["oid"]
|
||||
}'::jsonb
|
||||
);
|
||||
```
|
||||
|
||||
#### 模板 D：POST + JSON Body（腾讯音频风格，无 prefetch，无增量）
|
||||
```sql
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = '{平台编码}'),
|
||||
'{接口名称}', '{接口编码}',
|
||||
'{相对路径}', 'POST', 'active', 'inherit',
|
||||
'{
|
||||
"page": 1,
|
||||
"page_size": 100,
|
||||
"page_param": "page",
|
||||
"page_size_param": "page_size",
|
||||
"fields": ["field1", "field2"]
|
||||
}'::jsonb,
|
||||
'{
|
||||
"success_field": "code",
|
||||
"success_value": 0,
|
||||
"message_field": "message",
|
||||
"list_path": "data.list"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "{表名}",
|
||||
"columns": [
|
||||
{"name": "id", "type": "BIGINT", "comment": "ID"},
|
||||
{"name": "name", "type": "VARCHAR(200)", "comment": "名称"}
|
||||
],
|
||||
"conflict_keys": ["id"]
|
||||
}'::jsonb
|
||||
);
|
||||
```
|
||||
注意：POST 方法且无 `parameters_location: "query"` 时，参数以 JSON body 形式发送。
|
||||
|
||||
## 新增接口步骤总结
|
||||
|
||||
1. **确定平台**：已有平台直接跳到第3步，否则先新增平台（参考对应认证类型的模板）
|
||||
2. **确定平台 auth_type**：OAUTH2 / TOKEN / API_KEY / SIGN → 复制对应模板的 auth_config
|
||||
3. **分析 API**：
|
||||
- 请求方式（GET/POST）
|
||||
- 分页方式（普通分页 → page + total_page / 游标分页 → cursor + cursor_end）
|
||||
- 成功标识（code=0 / result=1 / errcode=0 等）
|
||||
- 数据列表路径（data.list / data.orderList / data.records 等）
|
||||
- 数据字段名（注意大小写，嵌套结构会被展开）
|
||||
- 是否需要遍历实体（prefetch）
|
||||
- 增量时间字段（字段名 + 值类型）
|
||||
4. **选择模板**并填入对应值
|
||||
5. **执行 seed SQL**
|
||||
6. **在管理后台验证**：`http://{host}:3002/admin`
|
||||
|
||||
@@ -16,7 +16,7 @@ sync:
|
||||
retry_count: 3 # 最大重试次数
|
||||
sync_interval_minutes: 60 # 自动同步间隔（分钟）
|
||||
compensation_interval_seconds: 300 # 补偿调度器扫描间隔（秒）
|
||||
auto_sync_enabled: true # 是否启用自动同步
|
||||
auto_sync_enabled: false # 是否启用自动同步
|
||||
|
||||
# Database.
|
||||
database:
|
||||
|
||||
355
controller/debug/admin_controller.go
Normal file
355
controller/debug/admin_controller.go
Normal file
@@ -0,0 +1,355 @@
|
||||
package debug
|
||||
|
||||
import (
|
||||
"github.com/gogf/gf/v2/net/ghttp"
|
||||
)
|
||||
|
||||
var DebugController = new(debugCtrl)
|
||||
|
||||
type debugCtrl struct{}
|
||||
|
||||
// DebugPage 调试页面
|
||||
func (c *debugCtrl) DebugPage(r *ghttp.Request) {
|
||||
r.Response.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
r.Response.Write(adminHTML)
|
||||
r.Exit()
|
||||
}
|
||||
|
||||
var adminHTML = `<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>数据引擎管理后台</title>
|
||||
<style>
|
||||
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||||
body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; background: #f0f2f5; color: #333; }
|
||||
.header { background: #001529; color: #fff; padding: 14px 24px; display: flex; align-items: center; gap: 12px; }
|
||||
.header h1 { font-size: 18px; font-weight: 600; }
|
||||
.container { max-width: 1400px; margin: 0 auto; padding: 20px; }
|
||||
.tabs { display: flex; gap: 0; margin-bottom: 20px; background: #fff; border-radius: 8px; overflow: hidden; box-shadow: 0 1px 4px rgba(0,0,0,0.06); }
|
||||
.tab { padding: 12px 28px; cursor: pointer; font-size: 14px; color: #666; border-bottom: 2px solid transparent; transition: all .2s; }
|
||||
.tab:hover { color: #1890ff; }
|
||||
.tab.active { color: #1890ff; border-bottom-color: #1890ff; font-weight: 600; }
|
||||
.panel { display: none; }
|
||||
.panel.active { display: block; }
|
||||
.toolbar { display: flex; justify-content: space-between; align-items: center; margin-bottom: 16px; }
|
||||
.toolbar h2 { font-size: 16px; color: #1a1a2e; }
|
||||
.btn { display: inline-flex; align-items: center; gap: 4px; padding: 8px 16px; border: none; border-radius: 6px; font-size: 13px; cursor: pointer; transition: all .2s; }
|
||||
.btn-primary { background: #1890ff; color: #fff; }
|
||||
.btn-primary:hover { background: #40a9ff; }
|
||||
.btn-danger { background: #ff4d4f; color: #fff; }
|
||||
.btn-danger:hover { background: #ff7875; }
|
||||
.btn-sm { padding: 4px 10px; font-size: 12px; }
|
||||
.btn-outline { background: #fff; border: 1px solid #d9d9d9; color: #333; }
|
||||
.btn-outline:hover { border-color: #1890ff; color: #1890ff; }
|
||||
.btn-success { background: #52c41a; color: #fff; }
|
||||
.btn-success:hover { background: #73d13d; }
|
||||
table { width: 100%; background: #fff; border-radius: 8px; overflow: hidden; box-shadow: 0 1px 4px rgba(0,0,0,0.06); border-collapse: collapse; }
|
||||
th { background: #fafafa; padding: 12px 16px; text-align: left; font-size: 13px; color: #666; font-weight: 600; border-bottom: 1px solid #f0f0f0; }
|
||||
td { padding: 12px 16px; font-size: 13px; border-bottom: 1px solid #f0f0f0; }
|
||||
tr:hover td { background: #fafafa; }
|
||||
.badge { display: inline-block; padding: 2px 8px; border-radius: 10px; font-size: 11px; font-weight: 500; }
|
||||
.badge.active { background: #f6ffed; color: #52c41a; border: 1px solid #b7eb8f; }
|
||||
.badge.inactive { background: #fff2f0; color: #ff4d4f; border: 1px solid #ffccc7; }
|
||||
.actions { display: flex; gap: 6px; }
|
||||
.modal-overlay { display: none; position: fixed; top: 0; left: 0; right: 0; bottom: 0; background: rgba(0,0,0,0.45); z-index: 1000; }
|
||||
.modal-overlay.open { display: block; }
|
||||
.modal { display: none; position: fixed; top: 50%; left: 50%; transform: translate(-50%,-50%); background: #fff; border-radius: 8px; width: 600px; max-width: 90%; max-height: 85vh; overflow-y: auto; z-index: 1001; box-shadow: 0 4px 24px rgba(0,0,0,0.15); }
|
||||
.modal.open { display: block; }
|
||||
.modal-header { padding: 16px 20px; border-bottom: 1px solid #f0f0f0; display: flex; justify-content: space-between; align-items: center; }
|
||||
.modal-header h3 { font-size: 16px; }
|
||||
.modal-close { background: none; border: none; font-size: 20px; cursor: pointer; color: #999; }
|
||||
.modal-body { padding: 20px; }
|
||||
.modal-footer { padding: 12px 20px; border-top: 1px solid #f0f0f0; display: flex; justify-content: flex-end; gap: 8px; }
|
||||
.form-group { margin-bottom: 14px; }
|
||||
.form-group label { display: block; font-size: 13px; color: #666; margin-bottom: 4px; font-weight: 500; }
|
||||
.form-group input, .form-group select, .form-group textarea { width: 100%; padding: 8px 12px; border: 1px solid #d9d9d9; border-radius: 6px; font-size: 13px; outline: none; transition: border .2s; }
|
||||
.form-group input:focus, .form-group select:focus { border-color: #1890ff; box-shadow: 0 0 0 2px rgba(24,144,255,0.1); }
|
||||
.form-group textarea { font-family: "Fira Code", "Consolas", monospace; font-size: 12px; min-height: 80px; resize: vertical; }
|
||||
.form-row { display: grid; grid-template-columns: 1fr 1fr; gap: 12px; }
|
||||
.loading { text-align: center; padding: 40px; color: #999; }
|
||||
.empty { text-align: center; padding: 32px; color: #999; font-size: 14px; }
|
||||
.interfaces-section { margin-top: 8px; }
|
||||
.sub-table { background: #fafafa; border-radius: 6px; overflow: hidden; margin-bottom: 8px; box-shadow: none; }
|
||||
.sub-table th { background: #f0f0f0; font-size: 12px; padding: 8px 12px; }
|
||||
.sub-table td { padding: 8px 12px; font-size: 12px; }
|
||||
.json-preview { max-width: 200px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; cursor: pointer; color: #1890ff; }
|
||||
.json-preview:hover { text-decoration: underline; }
|
||||
.platform-row { cursor: pointer; }
|
||||
.platform-row td:first-child { position: relative; }
|
||||
.expand-icon { margin-right: 8px; font-size: 10px; color: #999; transition: transform .2s; display: inline-block; }
|
||||
.expand-icon.open { transform: rotate(90deg); }
|
||||
.interface-detail { display: none; }
|
||||
.interface-detail.open { display: table-row; }
|
||||
.interface-detail td { padding: 0; }
|
||||
.interface-detail-inner { padding: 12px 16px 12px 44px; background: #fafafa; }
|
||||
.toast { position: fixed; top: 20px; right: 20px; padding: 12px 20px; border-radius: 6px; color: #fff; font-size: 13px; z-index: 2000; animation: slideIn .3s; }
|
||||
.toast.success { background: #52c41a; }
|
||||
.toast.error { background: #ff4d4f; }
|
||||
@keyframes slideIn { from { transform: translateX(100%); opacity: 0; } to { transform: translateX(0); opacity: 1; } }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="header"><h1>📊 数据引擎管理后台</h1></div>
|
||||
<div class="container">
|
||||
<div class="tabs">
|
||||
<div class="tab active" onclick="switchTab('platforms')">平台管理</div>
|
||||
<div class="tab" onclick="switchTab('interfaces')">接口管理</div>
|
||||
</div>
|
||||
|
||||
<div id="platforms-panel" class="panel active">
|
||||
<div class="toolbar"><h2>平台列表</h2><button class="btn btn-primary" onclick="openPlatformModal()">+ 新建平台</button></div>
|
||||
<div id="platform-loading" class="loading">加载中...</div>
|
||||
<div id="platform-list"></div>
|
||||
</div>
|
||||
|
||||
<div id="interfaces-panel" class="panel">
|
||||
<div class="toolbar"><h2>接口列表</h2><button class="btn btn-primary" onclick="openInterfaceModal()">+ 新建接口</button></div>
|
||||
<div id="interface-loading" class="loading">加载中...</div>
|
||||
<div id="interface-list"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="modal-overlay" id="modalOverlay" onclick="closeModal()"></div>
|
||||
<div class="modal" id="modal">
|
||||
<div class="modal-header"><h3 id="modalTitle"></h3><button class="modal-close" onclick="closeModal()">×</button></div>
|
||||
<div class="modal-body" id="modalBody"></div>
|
||||
<div class="modal-footer" id="modalFooter"></div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const BASE = '';
|
||||
|
||||
function toast(msg, type) {
|
||||
const t = document.createElement('div');
|
||||
t.className = 'toast ' + type;
|
||||
t.textContent = msg;
|
||||
document.body.appendChild(t);
|
||||
setTimeout(() => t.remove(), 2500);
|
||||
}
|
||||
|
||||
async function api(method, url, body) {
|
||||
const opt = { method, headers: {} };
|
||||
if (body) { opt.headers['Content-Type'] = 'application/json'; opt.body = JSON.stringify(body); }
|
||||
const resp = await fetch(BASE + url, opt);
|
||||
if (!resp.ok) { const t = await resp.text(); throw new Error(t || resp.statusText); }
|
||||
const json = await resp.json();
|
||||
// 解包 GoFrame 标准响应: {"code":0,"message":"success","data":{...}}
|
||||
if (json.code !== undefined && json.data !== undefined) {
|
||||
if (json.code !== 0) { throw new Error(json.message || '请求失败'); }
|
||||
return json.data;
|
||||
}
|
||||
return json;
|
||||
}
|
||||
|
||||
const P = { platform: '/datasource/platform/controller', iface: '/api/interface/controller' };
|
||||
|
||||
function switchTab(name) {
|
||||
document.querySelectorAll('.tab').forEach(t => t.classList.remove('active'));
|
||||
document.querySelectorAll('.panel').forEach(p => p.classList.remove('active'));
|
||||
document.querySelector('.tab[onclick*="' + name + '"]').classList.add('active');
|
||||
document.getElementById(name + '-panel').classList.add('active');
|
||||
if (name === 'platforms') loadPlatforms();
|
||||
else loadInterfaces();
|
||||
}
|
||||
|
||||
// ========== 平台管理 ==========
|
||||
async function loadPlatforms() {
|
||||
document.getElementById('platform-loading').style.display = 'block';
|
||||
try {
|
||||
const data = await api('GET', P.platform + '/listDatasourcePlatforms?page=1&pageSize=100');
|
||||
document.getElementById('platform-loading').style.display = 'none';
|
||||
if (!data.list || data.list.length === 0) {
|
||||
document.getElementById('platform-list').innerHTML = '<div class="empty">暂无平台，点击"新建平台"添加</div>';
|
||||
return;
|
||||
}
|
||||
let html = '<table><thead><tr><th style="width:30px"></th><th>编码</th><th>名称</th><th>API地址</th><th>认证</th><th>状态</th><th>操作</th></tr></thead><tbody>';
|
||||
for (const p of data.list) {
|
||||
const sc = p.status === 'ACTIVE' ? 'active' : 'inactive';
|
||||
html += '<tr class="platform-row" onclick="togglePlatform(' + p.id + ')">' +
|
||||
'<td><span class="expand-icon" id="expand-' + p.id + '">▶</span></td>' +
|
||||
'<td><strong>' + esc(p.platformCode) + '</strong></td>' +
|
||||
'<td>' + esc(p.platformName) + '</td>' +
|
||||
'<td style="font-size:12px;color:#666">' + esc(p.apiBaseUrl || '-') + '</td>' +
|
||||
'<td>' + esc(p.authType || '-') + '</td>' +
|
||||
'<td><span class="badge ' + sc + '">' + p.status + '</span></td>' +
|
||||
'<td class="actions" onclick="event.stopPropagation()">' +
|
||||
'<button class="btn btn-sm btn-outline" onclick="openPlatformModal(' + p.id + ')">编辑</button>' +
|
||||
'<button class="btn btn-sm btn-danger" onclick="deletePlatform(' + p.id + ')">删除</button></td></tr>' +
|
||||
'<tr class="interface-detail" id="ifaces-' + p.id + '"><td colspan="7"><div class="interface-detail-inner" id="ifaces-content-' + p.id + '">点击展开...</div></td></tr>';
|
||||
}
|
||||
html += '</tbody></table>';
|
||||
document.getElementById('platform-list').innerHTML = html;
|
||||
} catch (e) { document.getElementById('platform-loading').style.display = 'none'; toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
async function togglePlatform(id) {
|
||||
const row = document.getElementById('ifaces-' + id);
|
||||
const icon = document.getElementById('expand-' + id);
|
||||
row.classList.toggle('open');
|
||||
icon.classList.toggle('open');
|
||||
if (row.classList.contains('open') && document.getElementById('ifaces-content-' + id).textContent === '点击展开...') {
|
||||
try {
|
||||
const data = await api('GET', P.iface + '/listApiInterfaces?platformId=' + id + '&page=1&pageSize=100');
|
||||
const content = document.getElementById('ifaces-content-' + id);
|
||||
if (!data.list || data.list.length === 0) { content.innerHTML = '<div class="empty" style="padding:8px">暂无接口</div>'; return; }
|
||||
let h = '<table class="sub-table"><thead><tr><th>编码</th><th>名称</th><th>URL</th><th>方法</th><th>状态</th><th>请求配置</th><th>响应配置</th><th>操作</th></tr></thead><tbody>';
|
||||
for (const fi of data.list) {
|
||||
h += '<tr><td>' + esc(fi.code) + '</td><td>' + esc(fi.name) + '</td><td style="font-size:11px">' + esc(fi.url) + '</td>' +
|
||||
'<td><span class="badge ' + (fi.method === 'GET' ? 'active' : '') + '">' + fi.method + '</span></td>' +
|
||||
'<td><span class="badge ' + (fi.status === 'active' ? 'active' : 'inactive') + '">' + fi.status + '</span></td>' +
|
||||
'<td class="json-preview" title="' + esc(JSON.stringify(fi.requestConfig||{})) + '">' + esc(JSON.stringify(fi.requestConfig||{}).substring(0,25)) + '</td>' +
|
||||
'<td class="json-preview" title="' + esc(JSON.stringify(fi.responseConfig||{})) + '">' + esc(JSON.stringify(fi.responseConfig||{}).substring(0,25)) + '</td>' +
|
||||
'<td class="actions"><button class="btn btn-sm btn-outline" onclick="event.stopPropagation();openInterfaceModal(' + fi.id + ')">编辑</button>' +
|
||||
'<button class="btn btn-sm btn-danger" onclick="event.stopPropagation();deleteInterface(' + fi.id + ')">删</button></td></tr>';
|
||||
}
|
||||
h += '</tbody></table>';
|
||||
content.innerHTML = h;
|
||||
} catch (e) { document.getElementById('ifaces-content-' + id).textContent = '加载失败'; }
|
||||
}
|
||||
}
|
||||
|
||||
async function deletePlatform(id) {
|
||||
if (!confirm('确定删除此平台？')) return;
|
||||
try { await api('DELETE', P.platform + '/deleteDatasourcePlatform?id=' + id); toast('删除成功', 'success'); loadPlatforms(); }
|
||||
catch (e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
// ========== 接口管理 ==========
|
||||
async function loadInterfaces() {
|
||||
document.getElementById('interface-loading').style.display = 'block';
|
||||
try {
|
||||
const data = await api('GET', P.iface + '/listApiInterfaces?page=1&pageSize=100');
|
||||
document.getElementById('interface-loading').style.display = 'none';
|
||||
if (!data.list || data.list.length === 0) {
|
||||
document.getElementById('interface-list').innerHTML = '<div class="empty">暂无接口，点击"新建接口"添加</div>';
|
||||
return;
|
||||
}
|
||||
let html = '<table><thead><tr><th>编码</th><th>名称</th><th>URL</th><th>方法</th><th>状态</th><th>平台</th><th>请求配置</th><th>响应配置</th><th>操作</th></tr></thead><tbody>';
|
||||
for (const fi of data.list) {
|
||||
html += '<tr><td><strong>' + esc(fi.code) + '</strong></td><td>' + esc(fi.name) + '</td>' +
|
||||
'<td style="font-size:12px;color:#666">' + esc(fi.url) + '</td>' +
|
||||
'<td><span class="badge ' + (fi.method === 'GET' ? 'active' : '') + '">' + fi.method + '</span></td>' +
|
||||
'<td><span class="badge ' + (fi.status === 'active' ? 'active' : 'inactive') + '">' + fi.status + '</span></td>' +
|
||||
'<td>' + esc(fi.platformName || '-') + '</td>' +
|
||||
'<td class="json-preview" title="' + esc(JSON.stringify(fi.requestConfig||{})) + '">' + esc(JSON.stringify(fi.requestConfig||{}).substring(0,30)) + '</td>' +
|
||||
'<td class="json-preview" title="' + esc(JSON.stringify(fi.responseConfig||{})) + '">' + esc(JSON.stringify(fi.responseConfig||{}).substring(0,30)) + '</td>' +
|
||||
'<td class="actions"><button class="btn btn-sm btn-outline" onclick="openInterfaceModal(' + fi.id + ')">编辑</button>' +
|
||||
'<button class="btn btn-sm btn-danger" onclick="deleteInterface(' + fi.id + ')">删除</button></td></tr>';
|
||||
}
|
||||
html += '</tbody></table>';
|
||||
document.getElementById('interface-list').innerHTML = html;
|
||||
} catch (e) { document.getElementById('interface-loading').style.display = 'none'; toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
async function deleteInterface(id) {
|
||||
if (!confirm('确定删除此接口？')) return;
|
||||
try { await api('DELETE', P.iface + '/deleteApiInterface?id=' + id); toast('删除成功', 'success'); loadInterfaces(); loadPlatforms(); }
|
||||
catch (e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
// ========== 平台表单 ==========
|
||||
async function openPlatformModal(id) {
|
||||
const title = document.getElementById('modalTitle');
|
||||
const body = document.getElementById('modalBody');
|
||||
const footer = document.getElementById('modalFooter');
|
||||
if (id) {
|
||||
const data = await api('GET', P.platform + '/getDatasourcePlatform?id=' + id);
|
||||
title.textContent = '编辑平台';
|
||||
body.innerHTML = pfHTML(data);
|
||||
footer.innerHTML = '<button class="btn btn-outline" onclick="closeModal()">取消</button><button class="btn btn-primary" onclick="saveP(' + id + ')">保存</button>';
|
||||
} else {
|
||||
title.textContent = '新建平台';
|
||||
body.innerHTML = pfHTML(null);
|
||||
footer.innerHTML = '<button class="btn btn-outline" onclick="closeModal()">取消</button><button class="btn btn-primary" onclick="saveP(0)">创建</button>';
|
||||
}
|
||||
document.getElementById('modalOverlay').classList.add('open');
|
||||
document.getElementById('modal').classList.add('open');
|
||||
}
|
||||
function pfHTML(d) {
|
||||
d = d || {};
|
||||
const authCfg = d.authConfig ? JSON.stringify(d.authConfig, null, 2) : '{}';
|
||||
return '<div class="form-row"><div class="form-group"><label>编码 *</label><input id="f-pcode" value="' + esc(d.platformCode||'') + '"></div>' +
|
||||
'<div class="form-group"><label>名称 *</label><input id="f-pname" value="' + esc(d.platformName||'') + '"></div></div>' +
|
||||
'<div class="form-group"><label>API地址</label><input id="f-apiurl" value="' + esc(d.apiBaseUrl||'') + '"></div>' +
|
||||
'<div class="form-row"><div class="form-group"><label>认证类型 *</label><select id="f-auth"><option value="OAUTH2"' + (d.authType==='OAUTH2'?' selected':'') + '>OAuth2</option><option value="TOKEN"' + (d.authType==='TOKEN'?' selected':'') + '>Token</option><option value="API_KEY"' + (d.authType==='API_KEY'?' selected':'') + '>API Key</option><option value="SIGN"' + (d.authType==='SIGN'?' selected':'') + '>签名</option></select></div>' +
|
||||
'<div class="form-group"><label>状态</label><select id="f-ps"><option value="ACTIVE"' + ((d.status||'ACTIVE')==='ACTIVE'?' selected':'') + '>启用</option><option value="INACTIVE"' + (d.status==='INACTIVE'?' selected':'') + '>停用</option></select></div></div>' +
|
||||
'<div class="form-group"><label>Token / access_token</label><input id="f-tk" value="' + esc(d.token||'') + '"></div>' +
|
||||
'<div class="form-row"><div class="form-group"><label>Client ID</label><input id="f-cid" value="' + esc(d.clientId||'') + '"></div>' +
|
||||
'<div class="form-group"><label>Client Secret</label><input id="f-cs" value="' + esc(d.clientSecret||'') + '"></div></div>' +
|
||||
'<div class="form-group"><label>认证配置 JSON</label><textarea id="f-ac" rows="4">' + esc(authCfg) + '</textarea></div>' +
|
||||
'<div class="form-row"><div class="form-group"><label>限流/分钟</label><input id="f-rpm" type="number" value="' + (d.rateLimitPerMinute||60) + '"></div>' +
|
||||
'<div class="form-group"><label>超时(ms)</label><input id="f-to" type="number" value="' + (d.requestTimeoutMs||30000) + '"></div></div>';
|
||||
}
|
||||
async function saveP(id) {
|
||||
const now = Date.now().toString();
|
||||
const body = {
|
||||
platformCode: v('f-pcode'), platformName: v('f-pname'), apiBaseUrl: v('f-apiurl'),
|
||||
authType: v('f-auth'), status: v('f-ps'), token: v('f-tk'),
|
||||
clientId: v('f-cid'), clientSecret: v('f-cs'),
|
||||
rateLimitPerMinute: parseInt(v('f-rpm'))||60, requestTimeoutMs: parseInt(v('f-to'))||30000,
|
||||
createdBy: 'admin', createdAt: now, updatedBy: 'admin', updatedAt: now, version: '0',
|
||||
};
|
||||
try { body.authConfig = JSON.parse(v('f-ac')); } catch(e) { toast('auth_config JSON 格式错误', 'error'); return; }
|
||||
try {
|
||||
if (id) { await api('PUT', P.platform + '/updateDatasourcePlatform', { ...body, id }); toast('更新成功', 'success'); }
|
||||
else { await api('POST', P.platform + '/createDatasourcePlatform', body); toast('创建成功', 'success'); }
|
||||
closeModal(); loadPlatforms();
|
||||
} catch(e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
// ========== 接口表单 ==========
|
||||
async function openInterfaceModal(id) {
|
||||
const pl = await api('GET', P.platform + '/listDatasourcePlatforms?page=1&pageSize=100');
|
||||
const opts = (pl.list||[]).map(p => '<option value="' + p.id + '">' + esc(p.platformName) + '</option>').join('');
|
||||
const title = document.getElementById('modalTitle');
|
||||
const body = document.getElementById('modalBody');
|
||||
const footer = document.getElementById('modalFooter');
|
||||
if (id) {
|
||||
const data = await api('GET', P.iface + '/getApiInterface?id=' + id);
|
||||
title.textContent = '编辑接口';
|
||||
body.innerHTML = iFormHTML(data, opts);
|
||||
footer.innerHTML = '<button class="btn btn-outline" onclick="closeModal()">取消</button><button class="btn btn-primary" onclick="saveI(' + id + ')">保存</button>';
|
||||
} else {
|
||||
title.textContent = '新建接口';
|
||||
body.innerHTML = iFormHTML(null, opts);
|
||||
footer.innerHTML = '<button class="btn btn-outline" onclick="closeModal()">取消</button><button class="btn btn-primary" onclick="saveI(0)">创建</button>';
|
||||
}
|
||||
document.getElementById('modalOverlay').classList.add('open');
|
||||
document.getElementById('modal').classList.add('open');
|
||||
}
|
||||
function iFormHTML(d, opts) {
|
||||
d = d || {};
|
||||
const sel = opts.replace('value="' + (d.platformId||'') + '"', 'value="' + (d.platformId||'') + '" selected');
|
||||
return '<div class="form-group"><label>所属平台</label><select id="f-pid">' + sel + '</select></div>' +
|
||||
'<div class="form-row"><div class="form-group"><label>编码 *</label><input id="f-ic" value="' + esc(d.code||'') + '"></div>' +
|
||||
'<div class="form-group"><label>名称 *</label><input id="f-in" value="' + esc(d.name||'') + '"></div></div>' +
|
||||
'<div class="form-row"><div class="form-group"><label>URL *</label><input id="f-iu" value="' + esc(d.url||'') + '"></div>' +
|
||||
'<div class="form-group"><label>方法 *</label><select id="f-im"><option value="GET"' + ((d.method||'GET')==='GET'?' selected':'') + '>GET</option><option value="POST"' + (d.method==='POST'?' selected':'') + '>POST</option></select></div></div>' +
|
||||
'<div class="form-group"><label>请求配置 JSON</label><textarea id="f-rqc" rows="6">' + esc(d.requestConfig ? JSON.stringify(d.requestConfig, null, 2) : '{}') + '</textarea></div>' +
|
||||
'<div class="form-group"><label>响应配置 JSON</label><textarea id="f-rsc" rows="4">' + esc(d.responseConfig ? JSON.stringify(d.responseConfig, null, 2) : '{}') + '</textarea></div>' +
|
||||
'<div class="form-group"><label>表结构定义 JSON</label><textarea id="f-td" rows="6">' + esc(d.tableDefinition ? JSON.stringify(d.tableDefinition, null, 2) : '{}') + '</textarea></div>';
|
||||
}
|
||||
async function saveI(id) {
|
||||
let rqc, rsc, td;
|
||||
try { rqc = JSON.parse(v('f-rqc')); } catch(e) { toast('请求配置 JSON 错误', 'error'); return; }
|
||||
try { rsc = JSON.parse(v('f-rsc')); } catch(e) { toast('响应配置 JSON 错误', 'error'); return; }
|
||||
try { td = JSON.parse(v('f-td')); } catch(e) { toast('表结构定义 JSON 错误', 'error'); return; }
|
||||
const body = { platformId: parseInt(v('f-pid')), code: v('f-ic'), name: v('f-in'), url: v('f-iu'),
|
||||
method: v('f-im'), requestConfig: rqc, responseConfig: rsc, tableDefinition: td };
|
||||
try {
|
||||
if (id) { await api('PUT', P.iface + '/updateApiInterface', { ...body, id }); toast('更新成功', 'success'); }
|
||||
else { await api('POST', P.iface + '/createApiInterface', body); toast('创建成功', 'success'); }
|
||||
closeModal(); loadInterfaces(); loadPlatforms();
|
||||
} catch(e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
function closeModal() { document.getElementById('modalOverlay').classList.remove('open'); document.getElementById('modal').classList.remove('open'); }
|
||||
function v(id) { return document.getElementById(id).value; }
|
||||
function esc(s) { if (!s && s!==0) return ''; const d = document.createElement('div'); d.textContent = s; return d.innerHTML; }
|
||||
|
||||
loadPlatforms();
|
||||
</script>
|
||||
</body>
|
||||
</html>`
|
||||
@@ -15,25 +15,25 @@ var DatasourcePlatform = new(datasourcePlatformController)
|
||||
|
||||
// CreateDatasourcePlatform 创建数据源平台
|
||||
func (c *datasourcePlatformController) CreateDatasourcePlatform(ctx context.Context, req *dto.CreateDatasourcePlatformReq) (res *dto.CreateDatasourcePlatformRes, err error) {
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin"})
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1})
|
||||
return service.DatasourcePlatform.Create(ctx, req)
|
||||
}
|
||||
|
||||
// ListDatasourcePlatforms 获取数据源平台列表
|
||||
func (c *datasourcePlatformController) ListDatasourcePlatforms(ctx context.Context, req *dto.ListDatasourcePlatformReq) (res *dto.ListDatasourcePlatformRes, err error) {
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin"})
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1})
|
||||
return service.DatasourcePlatform.List(ctx, req)
|
||||
}
|
||||
|
||||
// GetDatasourcePlatform 获取数据源平台详情
|
||||
func (c *datasourcePlatformController) GetDatasourcePlatform(ctx context.Context, req *dto.GetDatasourcePlatformReq) (res *dto.GetDatasourcePlatformRes, err error) {
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin"})
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1})
|
||||
return service.DatasourcePlatform.GetOne(ctx, req)
|
||||
}
|
||||
|
||||
// GetPlatformByCode 根据平台编码获取平台信息
|
||||
func (c *datasourcePlatformController) GetPlatformByCode(ctx context.Context, req *dto.GetPlatformByCodeReq) (res *dto.GetPlatformByCodeRes, err error) {
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin"})
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1})
|
||||
platform, err := service.DatasourcePlatform.GetByPlatformCode(ctx, req.PlatformCode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -45,27 +45,27 @@ func (c *datasourcePlatformController) GetPlatformByCode(ctx context.Context, re
|
||||
|
||||
// UpdateDatasourcePlatform 更新数据源平台
|
||||
func (c *datasourcePlatformController) UpdateDatasourcePlatform(ctx context.Context, req *dto.UpdateDatasourcePlatformReq) (res *beans.ResponseEmpty, err error) {
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin"})
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1})
|
||||
err = service.DatasourcePlatform.Update(ctx, req)
|
||||
return
|
||||
}
|
||||
|
||||
// UpdateDatasourcePlatformStatus 更新数据源平台状态
|
||||
func (c *datasourcePlatformController) UpdateDatasourcePlatformStatus(ctx context.Context, req *dto.UpdateDatasourcePlatformStatusReq) (res *beans.ResponseEmpty, err error) {
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin"})
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1})
|
||||
err = service.DatasourcePlatform.UpdateStatus(ctx, req)
|
||||
return
|
||||
}
|
||||
|
||||
// DeleteDatasourcePlatform 删除数据源平台
|
||||
func (c *datasourcePlatformController) DeleteDatasourcePlatform(ctx context.Context, req *dto.DeleteDatasourcePlatformReq) (res *beans.ResponseEmpty, err error) {
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin"})
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1})
|
||||
err = service.DatasourcePlatform.Delete(ctx, req)
|
||||
return
|
||||
}
|
||||
|
||||
// GetPlatformStatistics 获取平台统计信息
|
||||
func (c *datasourcePlatformController) GetPlatformStatistics(ctx context.Context, req *dto.GetPlatformStatisticsReq) (res *dto.GetPlatformStatisticsRes, err error) {
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin"})
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1})
|
||||
return service.DatasourcePlatform.GetStatistics(ctx)
|
||||
}
|
||||
|
||||
@@ -15,39 +15,39 @@ var ApiInterface = new(apiInterfaceController)
|
||||
|
||||
// CreateApiInterface 创建接口
|
||||
func (c *apiInterfaceController) CreateApiInterface(ctx context.Context, req *dto.CreateApiInterfaceReq) (res *dto.CreateApiInterfaceRes, err error) {
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin"})
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1})
|
||||
return service.ApiInterface.Create(ctx, req)
|
||||
}
|
||||
|
||||
// ListApiInterface 获取接口列表
|
||||
func (c *apiInterfaceController) ListApiInterface(ctx context.Context, req *dto.ListApiInterfaceReq) (res *dto.ListApiInterfaceRes, err error) {
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin"})
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1})
|
||||
return service.ApiInterface.List(ctx, req)
|
||||
}
|
||||
|
||||
// GetApiInterface 获取接口详情
|
||||
func (c *apiInterfaceController) GetApiInterface(ctx context.Context, req *dto.GetApiInterfaceReq) (res *dto.GetApiInterfaceRes, err error) {
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin"})
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1})
|
||||
return service.ApiInterface.GetOne(ctx, req)
|
||||
}
|
||||
|
||||
// UpdateApiInterface 更新接口
|
||||
func (c *apiInterfaceController) UpdateApiInterface(ctx context.Context, req *dto.UpdateApiInterfaceReq) (res *beans.ResponseEmpty, err error) {
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin"})
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1})
|
||||
err = service.ApiInterface.Update(ctx, req)
|
||||
return
|
||||
}
|
||||
|
||||
// UpdateApiInterfaceStatus 更新接口状态
|
||||
func (c *apiInterfaceController) UpdateApiInterfaceStatus(ctx context.Context, req *dto.UpdateApiInterfaceStatusReq) (res *beans.ResponseEmpty, err error) {
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin"})
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1})
|
||||
err = service.ApiInterface.UpdateStatus(ctx, req)
|
||||
return
|
||||
}
|
||||
|
||||
// DeleteApiInterface 删除接口
|
||||
func (c *apiInterfaceController) DeleteApiInterface(ctx context.Context, req *dto.DeleteApiInterfaceReq) (res *beans.ResponseEmpty, err error) {
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin"})
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1})
|
||||
err = service.ApiInterface.Delete(ctx, req)
|
||||
return
|
||||
}
|
||||
|
||||
6
main.go
6
main.go
@@ -1,6 +1,7 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"dataengine/controller/debug"
|
||||
"dataengine/controller/dict"
|
||||
syncCtrl "dataengine/controller/sync"
|
||||
syncSvc "dataengine/service/sync"
|
||||
@@ -10,6 +11,7 @@ import (
|
||||
_ "gitea.com/red-future/common/k3sconfig"
|
||||
_ "github.com/gogf/gf/contrib/drivers/pgsql/v2"
|
||||
_ "github.com/gogf/gf/contrib/nosql/redis/v2"
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"golang.org/x/net/context"
|
||||
)
|
||||
|
||||
@@ -27,5 +29,9 @@ func main() {
|
||||
// 平台同步引擎
|
||||
syncCtrl.PlatformSyncController,
|
||||
})
|
||||
|
||||
// 管理后台页面
|
||||
g.Server().BindHandler("/admin", debug.DebugController.DebugPage)
|
||||
|
||||
select {}
|
||||
}
|
||||
|
||||
@@ -3,13 +3,16 @@ package sync
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/md5"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"math/big"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -24,8 +27,9 @@ type ApiResult struct {
|
||||
|
||||
// ApiClient 通用 API 客户端
|
||||
type ApiClient struct {
|
||||
config *PlatformConfig
|
||||
client *http.Client
|
||||
config *PlatformConfig
|
||||
client *http.Client
|
||||
rateLimiter <-chan time.Time // 限流 ticker
|
||||
}
|
||||
|
||||
// NewApiClient 创建客户端
|
||||
@@ -34,10 +38,17 @@ func NewApiClient(config *PlatformConfig) *ApiClient {
|
||||
if config.RequestTimeoutMs > 0 {
|
||||
timeout = time.Duration(config.RequestTimeoutMs) * time.Millisecond
|
||||
}
|
||||
return &ApiClient{
|
||||
ac := &ApiClient{
|
||||
config: config,
|
||||
client: &http.Client{Timeout: timeout},
|
||||
}
|
||||
// 初始化限流
|
||||
if config.RateLimitPerMinute > 0 {
|
||||
interval := time.Minute / time.Duration(config.RateLimitPerMinute)
|
||||
ac.rateLimiter = time.Tick(interval)
|
||||
logrus.Infof("限流已启用: %d 次/分钟, 间隔 %v", config.RateLimitPerMinute, interval)
|
||||
}
|
||||
return ac
|
||||
}
|
||||
|
||||
// Get 发送 GET 请求（无参数）
|
||||
@@ -85,6 +96,15 @@ func (c *ApiClient) doRequest(ctx context.Context, method, path string, body int
|
||||
}
|
||||
|
||||
func (c *ApiClient) execute(ctx context.Context, method, path string, body interface{}, paramsInQuery bool) (*ApiResult, error) {
|
||||
// 限流等待
|
||||
if c.rateLimiter != nil {
|
||||
select {
|
||||
case <-c.rateLimiter:
|
||||
case <-ctx.Done():
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
fullURL := c.config.GetApiUrl(path)
|
||||
|
||||
@@ -104,6 +124,8 @@ func (c *ApiClient) execute(ctx context.Context, method, path string, body inter
|
||||
}
|
||||
}
|
||||
|
||||
// 计算签名并追加（如快手 API 的 MD5 签名）
|
||||
fullURL = c.applySignature(fullURL, body, paramsInQuery)
|
||||
logrus.Infof("请求 URL: %s", fullURL)
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, method, fullURL, reqBody)
|
||||
@@ -192,6 +214,7 @@ func (c *ApiClient) applyAuthURL(rawURL string) string {
|
||||
for k, v := range eq {
|
||||
val := fmt.Sprintf("%v", v)
|
||||
val = strings.ReplaceAll(val, "{timestamp}", fmt.Sprintf("%d", time.Now().Unix()))
|
||||
val = strings.ReplaceAll(val, "{timestamp_ms}", fmt.Sprintf("%d", time.Now().UnixMilli()))
|
||||
val = strings.ReplaceAll(val, "{nonce}", generateNonce())
|
||||
extraParams[k] = val
|
||||
}
|
||||
@@ -250,3 +273,58 @@ func generateNonce() string {
|
||||
r, _ := rand.Int(rand.Reader, big.NewInt(10000))
|
||||
return fmt.Sprintf("%012d%04d", nanoPart, r.Int64())
|
||||
}
|
||||
|
||||
// applySignature 计算签名并追加到 URL（支持快手等平台的 MD5 签名）
|
||||
func (c *ApiClient) applySignature(rawURL string, body interface{}, paramsInQuery bool) string {
|
||||
cfg := c.config.AuthConfig
|
||||
if cfg == nil {
|
||||
return rawURL
|
||||
}
|
||||
|
||||
signAlgo, _ := cfg["sign_algorithm"].(string)
|
||||
if signAlgo == "" {
|
||||
return rawURL
|
||||
}
|
||||
appSecret, _ := cfg["app_secret"].(string)
|
||||
if appSecret == "" && c.config.AppSecret != "" {
|
||||
appSecret = c.config.AppSecret
|
||||
}
|
||||
if appSecret == "" {
|
||||
return rawURL
|
||||
}
|
||||
|
||||
parsed, _ := url.Parse(rawURL)
|
||||
q := parsed.Query()
|
||||
|
||||
// 收集所有参数并按 key 排序
|
||||
keys := make([]string, 0, len(q))
|
||||
for k := range q {
|
||||
if k == "sign" {
|
||||
continue
|
||||
}
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
|
||||
var signStr string
|
||||
for _, k := range keys {
|
||||
signStr += k + "=" + q.Get(k) + "&"
|
||||
}
|
||||
signStr += "key=" + appSecret
|
||||
|
||||
var sign string
|
||||
switch signAlgo {
|
||||
case "md5":
|
||||
h := md5.Sum([]byte(signStr))
|
||||
sign = hex.EncodeToString(h[:])
|
||||
case "md5_upper":
|
||||
h := md5.Sum([]byte(signStr))
|
||||
sign = strings.ToUpper(hex.EncodeToString(h[:]))
|
||||
default:
|
||||
return rawURL
|
||||
}
|
||||
|
||||
q.Set("sign", sign)
|
||||
parsed.RawQuery = q.Encode()
|
||||
return parsed.String()
|
||||
}
|
||||
|
||||
@@ -17,6 +17,9 @@ import (
|
||||
"github.com/sirupsen/logrus"
|
||||
)
|
||||
|
||||
// syncRunningMap 防止同一个接口被并发执行同步
|
||||
var syncRunningMap sync.Map
|
||||
|
||||
// SyncResult 同步结果
|
||||
type SyncResult struct {
|
||||
TableName string
|
||||
@@ -37,6 +40,14 @@ type PrefetchConfig struct {
|
||||
|
||||
// SyncByConfig 执行同步
|
||||
func SyncByConfig(ctx context.Context, platformCode, interfaceCode string, isFullSync bool) (*SyncResult, error) {
|
||||
// 内存锁：防止同一个接口被并发执行（两个调度周期重叠时跳过）
|
||||
lockKey := platformCode + "/" + interfaceCode
|
||||
if _, loaded := syncRunningMap.LoadOrStore(lockKey, true); loaded {
|
||||
logrus.Warnf("接口 [%s] 正在同步中，跳过重复请求", lockKey)
|
||||
return nil, fmt.Errorf("接口 [%s] 正在同步中，跳过", lockKey)
|
||||
}
|
||||
defer syncRunningMap.Delete(lockKey)
|
||||
|
||||
start := time.Now()
|
||||
pm := &PlatformManager{}
|
||||
|
||||
@@ -87,7 +98,7 @@ func SyncByConfig(ctx context.Context, platformCode, interfaceCode string, isFul
|
||||
if prefetch != nil {
|
||||
return syncWithPrefetch(ctx, api, platform, iface, ifaces, td, prefetch, isFullSync, lastSyncTime, start)
|
||||
}
|
||||
return syncSingleAPI(ctx, api, platform, iface, td, lastSyncTime, start)
|
||||
return syncSingleAPI(ctx, api, platform, iface, td, isFullSync, lastSyncTime, start)
|
||||
}
|
||||
|
||||
// paramsInQuery 判断参数是否应放在 URL 查询字符串中
|
||||
@@ -104,24 +115,39 @@ func paramsInQuery(iface *entity.ApiInterface) bool {
|
||||
}
|
||||
|
||||
// syncSingleAPI 单接口分页同步
|
||||
func syncSingleAPI(ctx context.Context, api *ApiClient, platform *PlatformConfig, iface *entity.ApiInterface, td *TableDefinition, lastSyncTime int64, start time.Time) (*SyncResult, error) {
|
||||
func syncSingleAPI(ctx context.Context, api *ApiClient, platform *PlatformConfig, iface *entity.ApiInterface, td *TableDefinition, isFullSync bool, lastSyncTime int64, start time.Time) (*SyncResult, error) {
|
||||
pageSize := GetSyncPageSize(ctx)
|
||||
if ps, ok := iface.RequestConfig["page_size"].(float64); ok {
|
||||
pageSize = int(ps)
|
||||
}
|
||||
|
||||
taskType := "incremental"
|
||||
if isFullSync || lastSyncTime <= 0 {
|
||||
taskType = "full"
|
||||
}
|
||||
|
||||
inQuery := paramsInQuery(iface)
|
||||
method := string(iface.Method)
|
||||
|
||||
body := buildReqBody(iface, 1, pageSize, lastSyncTime, nil)
|
||||
// 游标分页首次请求需要 cursor=""（通过 extraParams 覆盖 buildReqBody 的 page=1 赋值）
|
||||
firstExtra := map[string]interface{}{}
|
||||
if isCursorPagination(iface) {
|
||||
cp := "cursor"
|
||||
if p, ok := iface.RequestConfig["page_param"].(string); ok && p != "" {
|
||||
cp = p
|
||||
}
|
||||
firstExtra[cp] = ""
|
||||
}
|
||||
body := buildReqBody(iface, 1, pageSize, lastSyncTime, firstExtra)
|
||||
resp, err := api.Request(ctx, method, iface.Url, body, inQuery)
|
||||
if err != nil {
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, err.Error())
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, err.Error())
|
||||
return nil, fmt.Errorf("获取第一页失败: %w", err)
|
||||
}
|
||||
|
||||
rows, totalPages, maxTime, err := parseResp(resp.Body, iface.ResponseConfig)
|
||||
rows, totalPages, maxTime, nextCursor, err := parseRespExt(resp.Body, iface.ResponseConfig)
|
||||
if err != nil {
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("解析第一页响应失败: %v", err))
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -130,24 +156,68 @@ func syncSingleAPI(ctx context.Context, api *ApiClient, platform *PlatformConfig
|
||||
result.InsertedRows += inserted
|
||||
result.TotalRows += len(rows)
|
||||
|
||||
for page := 2; page <= totalPages; page++ {
|
||||
body := buildReqBody(iface, page, pageSize, lastSyncTime, nil)
|
||||
resp, err := api.Request(ctx, method, iface.Url, body, inQuery)
|
||||
if err != nil {
|
||||
logrus.Errorf("第 %d 页失败: %v", page, err)
|
||||
continue
|
||||
// 游标分页
|
||||
if isCursorPagination(iface) {
|
||||
for nextCursor != "" && nextCursor != "nomore" {
|
||||
cp := "cursor"
|
||||
if p, ok := iface.RequestConfig["page_param"].(string); ok && p != "" {
|
||||
cp = p
|
||||
}
|
||||
body := buildReqBody(iface, 1, pageSize, lastSyncTime, map[string]interface{}{
|
||||
cp: nextCursor,
|
||||
})
|
||||
|
||||
resp, err := api.Request(ctx, method, iface.Url, body, inQuery)
|
||||
if err != nil {
|
||||
logrus.Errorf("游标 %s 请求失败: %v", nextCursor, err)
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("游标 %s 请求失败: %v", nextCursor, err))
|
||||
break
|
||||
}
|
||||
|
||||
rows, _, mt, nc, pe := parseRespExt(resp.Body, iface.ResponseConfig)
|
||||
if pe != nil {
|
||||
logrus.Errorf("游标 %s 解析失败: %v", nextCursor, pe)
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("游标 %s 解析失败: %v", nextCursor, pe))
|
||||
break
|
||||
}
|
||||
if len(rows) == 0 {
|
||||
break
|
||||
}
|
||||
nextCursor = nc
|
||||
|
||||
inserted, _ = savePage(ctx, td, rows)
|
||||
result.InsertedRows += inserted
|
||||
result.TotalRows += len(rows)
|
||||
if mt > maxTime {
|
||||
maxTime = mt
|
||||
}
|
||||
result.TotalPages++
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
rows, _, mt, err := parseResp(resp.Body, iface.ResponseConfig)
|
||||
if err != nil {
|
||||
continue
|
||||
} else {
|
||||
// 普通分页
|
||||
for page := 2; page <= totalPages; page++ {
|
||||
body := buildReqBody(iface, page, pageSize, lastSyncTime, nil)
|
||||
resp, err := api.Request(ctx, method, iface.Url, body, inQuery)
|
||||
if err != nil {
|
||||
logrus.Errorf("第 %d 页请求失败: %v", page, err)
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("第 %d 页请求失败: %v", page, err))
|
||||
continue
|
||||
}
|
||||
rows, _, mt, _, pe := parseRespExt(resp.Body, iface.ResponseConfig)
|
||||
if pe != nil {
|
||||
logrus.Errorf("第 %d 页解析失败: %v", page, pe)
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("第 %d 页解析失败: %v", page, pe))
|
||||
continue
|
||||
}
|
||||
inserted, _ = savePage(ctx, td, rows)
|
||||
result.InsertedRows += inserted
|
||||
result.TotalRows += len(rows)
|
||||
if mt > maxTime {
|
||||
maxTime = mt
|
||||
}
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
inserted, _ = savePage(ctx, td, rows)
|
||||
result.InsertedRows += inserted
|
||||
result.TotalRows += len(rows)
|
||||
if mt > maxTime {
|
||||
maxTime = mt
|
||||
}
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
|
||||
if maxTime <= 0 {
|
||||
@@ -160,84 +230,150 @@ func syncSingleAPI(ctx context.Context, api *ApiClient, platform *PlatformConfig
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func isCursorPagination(iface *entity.ApiInterface) bool {
|
||||
if iface.RequestConfig == nil {
|
||||
return false
|
||||
}
|
||||
cp, _ := iface.RequestConfig["cursor_pagination"].(bool)
|
||||
return cp
|
||||
}
|
||||
|
||||
// collectPrefetchEntities 从 rows 中收集实体和行数据
|
||||
func collectPrefetchEntities(rows []map[string]interface{}, prefetch *PrefetchConfig, allEntities *[]interface{}, allRows *[]map[string]interface{}) {
|
||||
for _, item := range rows {
|
||||
*allRows = append(*allRows, item)
|
||||
if prefetch.ValueField == "" {
|
||||
*allEntities = append(*allEntities, item)
|
||||
} else if v, ok := item[prefetch.ValueField]; ok {
|
||||
if f, ok := v.(float64); ok {
|
||||
*allEntities = append(*allEntities, int64(f))
|
||||
} else {
|
||||
*allEntities = append(*allEntities, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// syncWithPrefetch 预取模式同步（先分页拉取全部实体列表，再并发处理每个实体）
|
||||
func syncWithPrefetch(ctx context.Context, api *ApiClient, platform *PlatformConfig, iface *entity.ApiInterface, allIfaces []entity.ApiInterface, td *TableDefinition, prefetch *PrefetchConfig, isFullSync bool, lastSyncTime int64, start time.Time) (*SyncResult, error) {
|
||||
logrus.Infof("预取模式: %s -> %s", prefetch.URL, iface.Url)
|
||||
|
||||
// 1. 查找匹配 prefetch URL 的接口配置（用于获取正确的请求参数）
|
||||
taskType := "incremental"
|
||||
if isFullSync || lastSyncTime <= 0 {
|
||||
taskType = "full"
|
||||
}
|
||||
|
||||
// ====== 1. 预取阶段：分页拉取全部实体列表 ======
|
||||
prefetchIface := findInterfaceByURL(allIfaces, prefetch.URL)
|
||||
prefetchParams := buildPrefetchParams(iface)
|
||||
|
||||
// 判断预取来源是否游标分页，以及分页参数名
|
||||
prefetchIsCursor := false
|
||||
prefetchPageParam := "page"
|
||||
if prefetchIface != nil && prefetchIface.RequestConfig != nil {
|
||||
// 使用 prefetch 目标接口的 request_config 重建参数（覆盖默认值）
|
||||
for k, v := range prefetchIface.RequestConfig {
|
||||
if k == "headers" || k == "prefetch" || k == "page_param" ||
|
||||
k == "page_size_param" || k == "time_field" || k == "parameters_location" ||
|
||||
k == "filtering" || k == "group_by" || k == "date_range" {
|
||||
continue
|
||||
}
|
||||
prefetchParams[k] = v
|
||||
if cp, ok := prefetchIface.RequestConfig["cursor_pagination"].(bool); ok {
|
||||
prefetchIsCursor = cp
|
||||
}
|
||||
if p, ok := prefetchIface.RequestConfig["page_param"].(string); ok && p != "" {
|
||||
prefetchPageParam = p
|
||||
}
|
||||
}
|
||||
method := strings.ToUpper(prefetch.Method)
|
||||
inQuery := paramsInQuery(iface)
|
||||
|
||||
prefetchMethod := strings.ToUpper(prefetch.Method)
|
||||
prefetchPageSize := 100
|
||||
if prefetchIface != nil && prefetchIface.RequestConfig != nil {
|
||||
if ps, ok := prefetchIface.RequestConfig["pageSize"].(float64); ok {
|
||||
prefetchPageSize = int(ps)
|
||||
}
|
||||
}
|
||||
|
||||
// 使用 prefetch 来源接口自己的配置判断参数位置
|
||||
var prefetchInQuery bool
|
||||
if prefetchIface != nil {
|
||||
prefetchInQuery = paramsInQuery(prefetchIface)
|
||||
} else {
|
||||
prefetchInQuery = paramsInQuery(iface)
|
||||
}
|
||||
|
||||
// prefetch 来源接口的 response_config（用于正确解析列表路径）
|
||||
var prefetchRespCfg map[string]interface{}
|
||||
if prefetchIface != nil {
|
||||
prefetchRespCfg = prefetchIface.ResponseConfig
|
||||
}
|
||||
|
||||
allEntities := make([]interface{}, 0)
|
||||
allRows := make([]map[string]interface{}, 0)
|
||||
prefetchPage := 1
|
||||
prefetchTotalPages := 1
|
||||
|
||||
for prefetchPage <= prefetchTotalPages {
|
||||
params := make(map[string]interface{})
|
||||
for k, v := range prefetchParams {
|
||||
params[k] = v
|
||||
}
|
||||
pageParam := "page"
|
||||
if p, ok := iface.RequestConfig["page_param"].(string); ok {
|
||||
pageParam = p
|
||||
}
|
||||
params[pageParam] = prefetchPage
|
||||
// 第一页（游标分页首次 cursor=""）
|
||||
firstExtra := make(map[string]interface{})
|
||||
if prefetchIsCursor {
|
||||
firstExtra[prefetchPageParam] = ""
|
||||
}
|
||||
prefetchReqIface := prefetchIface
|
||||
if prefetchReqIface == nil {
|
||||
prefetchReqIface = iface
|
||||
}
|
||||
body := buildReqBody(prefetchReqIface, 1, prefetchPageSize, lastSyncTime, firstExtra)
|
||||
resp, err := api.Request(ctx, prefetchMethod, prefetch.URL, body, prefetchInQuery)
|
||||
if err != nil {
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("预取第一页请求失败: %v", err))
|
||||
return nil, fmt.Errorf("预取第一页失败: %w", err)
|
||||
}
|
||||
|
||||
resp, err := api.Request(ctx, method, prefetch.URL, params, true)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("预取第 %d 页失败: %w", prefetchPage, err)
|
||||
}
|
||||
rows, prefetchTotalPages, _, nextCursor, err := parseRespExt(resp.Body, prefetchRespCfg)
|
||||
if err != nil {
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("解析预取响应失败: %v", err))
|
||||
return nil, fmt.Errorf("解析预取响应失败: %w", err)
|
||||
}
|
||||
collectPrefetchEntities(rows, prefetch, &allEntities, &allRows)
|
||||
|
||||
entities, _, _, err := parseResp(resp.Body, nil)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("解析预取第 %d 页响应失败: %w", prefetchPage, err)
|
||||
}
|
||||
|
||||
// 收集完整数据行（用于存库）和提取的 ID 值（用于遍历）
|
||||
for _, item := range entities {
|
||||
allRows = append(allRows, item)
|
||||
if prefetch.ValueField == "" {
|
||||
allEntities = append(allEntities, item)
|
||||
} else if v, ok := item[prefetch.ValueField]; ok {
|
||||
// 将 float64 转 int64，避免后续 URL 参数中出现科学计数法
|
||||
if f, ok := v.(float64); ok {
|
||||
allEntities = append(allEntities, int64(f))
|
||||
} else {
|
||||
allEntities = append(allEntities, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if prefetchPage == 1 {
|
||||
if tp := getTotalPages(resp.Body); tp > 0 {
|
||||
prefetchTotalPages = tp
|
||||
} else {
|
||||
// 分页循环
|
||||
if prefetchIsCursor {
|
||||
// 游标分页
|
||||
for nextCursor != "" && nextCursor != "nomore" {
|
||||
body := buildReqBody(prefetchReqIface, 1, prefetchPageSize, lastSyncTime, map[string]interface{}{
|
||||
prefetchPageParam: nextCursor,
|
||||
})
|
||||
resp, err := api.Request(ctx, prefetchMethod, prefetch.URL, body, prefetchInQuery)
|
||||
if err != nil {
|
||||
logrus.Errorf("预取游标 %s 请求失败: %v", nextCursor, err)
|
||||
break
|
||||
}
|
||||
rows, _, _, nc, pe := parseRespExt(resp.Body, prefetchRespCfg)
|
||||
if pe != nil {
|
||||
logrus.Errorf("预取游标 %s 解析失败: %v", nextCursor, pe)
|
||||
break
|
||||
}
|
||||
if len(rows) == 0 {
|
||||
break
|
||||
}
|
||||
nextCursor = nc
|
||||
collectPrefetchEntities(rows, prefetch, &allEntities, &allRows)
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
} else {
|
||||
// 普通分页
|
||||
for page := 2; page <= prefetchTotalPages; page++ {
|
||||
body := buildReqBody(prefetchReqIface, page, prefetchPageSize, lastSyncTime, nil)
|
||||
resp, err := api.Request(ctx, prefetchMethod, prefetch.URL, body, prefetchInQuery)
|
||||
if err != nil {
|
||||
logrus.Errorf("预取第 %d 页请求失败: %v", page, err)
|
||||
continue
|
||||
}
|
||||
rows, _, _, _, pe := parseRespExt(resp.Body, prefetchRespCfg)
|
||||
if pe != nil {
|
||||
logrus.Errorf("预取第 %d 页解析失败: %v", page, pe)
|
||||
continue
|
||||
}
|
||||
collectPrefetchEntities(rows, prefetch, &allEntities, &allRows)
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
prefetchPage++
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
}
|
||||
|
||||
if len(allEntities) == 0 {
|
||||
logrus.Warn("预取结果为空列表，跳过同步")
|
||||
return &SyncResult{TableName: td.TableName, Duration: fmt.Sprintf("%.1fs", time.Since(start).Seconds())}, nil
|
||||
}
|
||||
logrus.Infof("预取到 %d 个实体（共 %d 页）", len(allEntities), prefetchPage-1)
|
||||
logrus.Infof("预取到 %d 个实体", len(allEntities))
|
||||
|
||||
// 2. 将预取的数据也存入库（如账户列表存入 tencent_account_relation）
|
||||
if prefetchIface != nil && prefetchIface.TableDefinition != nil {
|
||||
@@ -258,6 +394,7 @@ func syncWithPrefetch(ctx context.Context, api *ApiClient, platform *PlatformCon
|
||||
}
|
||||
|
||||
dataMethod := string(iface.Method)
|
||||
inQuery := paramsInQuery(iface)
|
||||
concurrency := GetSyncConcurrency(ctx)
|
||||
|
||||
var mu sync.Mutex
|
||||
@@ -346,21 +483,24 @@ func syncWithPrefetch(ctx context.Context, api *ApiClient, platform *PlatformCon
|
||||
|
||||
// getTotalPages 从响应中提取总页数
|
||||
func getTotalPages(raw []byte) int {
|
||||
var r struct {
|
||||
Data map[string]interface{} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &r); err != nil {
|
||||
rows, tp, _, _, err := parseRespExt(raw, nil)
|
||||
if err != nil || len(rows) == 0 {
|
||||
return 0
|
||||
}
|
||||
if r.Data == nil {
|
||||
return 0
|
||||
return tp
|
||||
}
|
||||
|
||||
func toFloat64(v interface{}) (float64, bool) {
|
||||
switch val := v.(type) {
|
||||
case float64:
|
||||
return val, true
|
||||
case int:
|
||||
return float64(val), true
|
||||
case int64:
|
||||
return float64(val), true
|
||||
default:
|
||||
return 0, false
|
||||
}
|
||||
if pi, ok := r.Data["page_info"].(map[string]interface{}); ok {
|
||||
if tp, ok := pi["total_page"].(float64); ok {
|
||||
return int(tp)
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// buildPrefetchParams 构建预取接口的请求参数
|
||||
@@ -382,7 +522,9 @@ func buildPrefetchParams(iface *entity.ApiInterface) map[string]interface{} {
|
||||
for k, v := range iface.RequestConfig {
|
||||
if k == "headers" || k == "prefetch" || k == "page_param" ||
|
||||
k == "page_size_param" || k == "time_field" || k == "parameters_location" ||
|
||||
k == "filtering" || k == "group_by" || k == "date_range" {
|
||||
k == "filtering" || k == "group_by" || k == "date_range" ||
|
||||
k == "body_wrapper_field" || k == "exclude_from_wrapper" ||
|
||||
k == "cursor_pagination" || k == "time_field_mode" {
|
||||
continue
|
||||
}
|
||||
if k == pageParam || k == psParam {
|
||||
@@ -467,7 +609,10 @@ func buildReqBody(iface *entity.ApiInterface, page, pageSize int, lastSyncTime i
|
||||
if iface.RequestConfig != nil {
|
||||
for k, v := range iface.RequestConfig {
|
||||
if k == "time_field" || k == "headers" || k == "prefetch" ||
|
||||
k == "page_param" || k == "page_size_param" || k == "parameters_location" {
|
||||
k == "page_param" || k == "page_size_param" || k == "parameters_location" ||
|
||||
k == "cursor_pagination" || k == "time_field_mode" ||
|
||||
k == "body_wrapper_field" || k == "exclude_from_wrapper" ||
|
||||
k == "top_level_params" {
|
||||
continue
|
||||
}
|
||||
body[k] = v
|
||||
@@ -485,16 +630,33 @@ func buildReqBody(iface *entity.ApiInterface, page, pageSize int, lastSyncTime i
|
||||
}
|
||||
body[pageParam] = page
|
||||
body[psParam] = pageSize
|
||||
// 增量同步：将 time_field 转为 API 期望的 filtering 格式
|
||||
// 如 filtering=[{"field":"last_modified_time","operator":"GREATER_EQUALS","values":["1780037982"]}]
|
||||
if lastSyncTime > 0 {
|
||||
if tf, ok := iface.RequestConfig["time_field"].(string); ok && tf != "" {
|
||||
|
||||
// 时间过滤处理：支持两种模式
|
||||
// 1. "filtering" 模式（默认）：生成 filtering=[{"field":"...","operator":"GREATER_EQUALS","values":["..."]}]（腾讯）
|
||||
// 2. "range" 模式：生成 beginTime/endTime + queryType（快手）
|
||||
if tf, ok := iface.RequestConfig["time_field"].(string); ok && tf != "" {
|
||||
timeMode := "filtering"
|
||||
if tm, ok := iface.RequestConfig["time_field_mode"].(string); ok && tm != "" {
|
||||
timeMode = tm
|
||||
}
|
||||
|
||||
if timeMode == "range" {
|
||||
// 快手模式：beginTime/endTime（毫秒时间戳）
|
||||
timeMs := lastSyncTime
|
||||
if timeMs <= 0 {
|
||||
// 全量：默认90天前
|
||||
timeMs = time.Now().Add(-90 * 24 * time.Hour).UnixMilli()
|
||||
}
|
||||
body["queryType"] = 2
|
||||
body["beginTime"] = timeMs
|
||||
body["endTime"] = time.Now().UnixMilli()
|
||||
} else if lastSyncTime > 0 {
|
||||
// 腾讯 filtering 模式（仅增量时）
|
||||
timeFilter := map[string]interface{}{
|
||||
"field": tf,
|
||||
"operator": "GREATER_EQUALS",
|
||||
"values": []interface{}{fmt.Sprintf("%d", lastSyncTime)},
|
||||
}
|
||||
// 合并已有的 filtering（如果 request_config 中已定义其他过滤条件）
|
||||
if existing, ok := body["filtering"].([]interface{}); ok {
|
||||
body["filtering"] = append(existing, timeFilter)
|
||||
} else {
|
||||
@@ -502,64 +664,197 @@ func buildReqBody(iface *entity.ApiInterface, page, pageSize int, lastSyncTime i
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for k, v := range extraParams {
|
||||
body[k] = v
|
||||
}
|
||||
|
||||
// body_wrapper_field 支持：将业务参数包装到指定字段（如快手 API 的 param JSON）
|
||||
if iface.RequestConfig != nil {
|
||||
if wf, ok := iface.RequestConfig["body_wrapper_field"].(string); ok && wf != "" {
|
||||
excludeSet := map[string]bool{"method": true, "version": true, "signMethod": true}
|
||||
if excl, ok := iface.RequestConfig["exclude_from_wrapper"].([]interface{}); ok {
|
||||
for _, v := range excl {
|
||||
if s, ok := v.(string); ok {
|
||||
excludeSet[s] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
wrapperObj := make(map[string]interface{})
|
||||
for k, v := range body {
|
||||
if !excludeSet[k] && k != wf {
|
||||
wrapperObj[k] = v
|
||||
delete(body, k)
|
||||
}
|
||||
}
|
||||
b, _ := json.Marshal(wrapperObj)
|
||||
body[wf] = string(b)
|
||||
}
|
||||
}
|
||||
|
||||
return body
|
||||
}
|
||||
|
||||
// parseResp 解析同步接口返回值
|
||||
func parseResp(raw []byte, responseConfig map[string]interface{}) ([]map[string]interface{}, int, int64, error) {
|
||||
var r struct {
|
||||
Code int `json:"code"`
|
||||
Message string `json:"message"`
|
||||
Data map[string]interface{} `json:"data"`
|
||||
// parseRespExt 解析响应，支持自定义成功判断和数据路径
|
||||
func parseRespExt(raw []byte, rc map[string]interface{}) ([]map[string]interface{}, int, int64, string, error) {
|
||||
var respMap map[string]interface{}
|
||||
if err := json.Unmarshal(raw, &respMap); err != nil {
|
||||
return nil, 0, 0, "", fmt.Errorf("JSON解析失败: %w", err)
|
||||
}
|
||||
if err := json.Unmarshal(raw, &r); err != nil {
|
||||
return nil, 0, 0, fmt.Errorf("解析响应失败: %w", err)
|
||||
}
|
||||
if r.Code != 0 {
|
||||
return nil, 0, 0, fmt.Errorf("API错误: code=%d, message=%s", r.Code, r.Message)
|
||||
}
|
||||
|
||||
var rows []map[string]interface{}
|
||||
totalPages := 1
|
||||
maxTime := int64(0)
|
||||
|
||||
var listData []interface{}
|
||||
if lp, ok := r.Data["list"]; ok {
|
||||
listData, _ = lp.([]interface{})
|
||||
} else if lp, ok := r.Data["data"]; ok {
|
||||
if m, ok := lp.(map[string]interface{}); ok {
|
||||
if l, ok := m["list"].([]interface{}); ok {
|
||||
listData = l
|
||||
successField, successVal := "code", float64(0)
|
||||
msgField, listPath, cursorPath := "message", "data", ""
|
||||
singleRecord := false
|
||||
if rc != nil {
|
||||
if sf, _ := rc["success_field"].(string); sf != "" {
|
||||
successField = sf
|
||||
}
|
||||
if sv, ok := rc["success_value"]; ok {
|
||||
if f, ok := toFloat64(sv); ok {
|
||||
successVal = f
|
||||
}
|
||||
}
|
||||
if mf, _ := rc["message_field"].(string); mf != "" {
|
||||
msgField = mf
|
||||
}
|
||||
if lp, _ := rc["list_path"].(string); lp != "" {
|
||||
listPath = lp
|
||||
}
|
||||
if cf, _ := rc["cursor_field"].(string); cf != "" {
|
||||
cursorPath = cf
|
||||
}
|
||||
if sr, _ := rc["single_record"].(bool); sr {
|
||||
singleRecord = true
|
||||
}
|
||||
}
|
||||
if v, ok := respMap[successField]; ok {
|
||||
actual, _ := toFloat64(v)
|
||||
if actual != successVal {
|
||||
msg, _ := respMap[msgField].(string)
|
||||
return nil, 0, 0, "", fmt.Errorf("API错误: %s=%v, %s=%s", successField, v, msgField, msg)
|
||||
}
|
||||
}
|
||||
|
||||
// 解析 list_path，支持最后一段是数组的情况（如 data.orderList）
|
||||
var listData []interface{}
|
||||
var dataContainer map[string]interface{}
|
||||
if listPath != "" {
|
||||
parts := strings.Split(listPath, ".")
|
||||
cur := respMap
|
||||
for i, p := range parts {
|
||||
if i == len(parts)-1 {
|
||||
// 最后一段：可能直接是数组，也可能是包含 list/orderList 的 map
|
||||
listData, _ = cur[p].([]interface{})
|
||||
if listData == nil {
|
||||
if m, ok := cur[p].(map[string]interface{}); ok {
|
||||
dataContainer = m
|
||||
if l, ok := m["list"]; ok {
|
||||
listData, _ = l.([]interface{})
|
||||
}
|
||||
if listData == nil {
|
||||
if ol, ok := m["orderList"]; ok {
|
||||
listData, _ = ol.([]interface{})
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
dataContainer = cur
|
||||
}
|
||||
} else {
|
||||
next, ok := cur[p].(map[string]interface{})
|
||||
if !ok {
|
||||
return nil, 0, 0, "", fmt.Errorf("路径 %s 在 %s 处中断", listPath, p)
|
||||
}
|
||||
cur = next
|
||||
}
|
||||
}
|
||||
}
|
||||
if listData == nil {
|
||||
if singleRecord && listPath != "" {
|
||||
// 详情接口：list_path 指向单个对象，包装为单元素数组
|
||||
parts := strings.Split(listPath, ".")
|
||||
cur := respMap
|
||||
ok := true
|
||||
for _, p := range parts {
|
||||
if m, exists := cur[p].(map[string]interface{}); exists {
|
||||
cur = m
|
||||
} else {
|
||||
ok = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if ok {
|
||||
listData = []interface{}{cur}
|
||||
dataContainer = cur
|
||||
}
|
||||
}
|
||||
}
|
||||
if listData == nil {
|
||||
// 回退到根级字段
|
||||
listData, _ = respMap["list"].([]interface{})
|
||||
if listData == nil {
|
||||
listData, _ = respMap["orderList"].([]interface{})
|
||||
}
|
||||
dataContainer = respMap
|
||||
}
|
||||
var rows []map[string]interface{}
|
||||
totalPages, maxTime := 1, int64(0)
|
||||
for _, item := range listData {
|
||||
if m, ok := item.(map[string]interface{}); ok {
|
||||
// 展平嵌套 map：将子 map 的字段合并到顶层（如 orderBaseInfo.oid → oid）
|
||||
flat := flattenRow(m)
|
||||
j, _ := json.Marshal(m)
|
||||
m["raw_data"] = string(j)
|
||||
if t, ok := m["last_modified_time"].(float64); ok && int64(t) > maxTime {
|
||||
maxTime = int64(t)
|
||||
flat["raw_data"] = string(j)
|
||||
for _, tf := range []string{"last_modified_time", "created_time", "update_time", "createTime", "updateTime", "lastModifiedTime"} {
|
||||
if t, ok := flat[tf].(float64); ok && int64(t) > maxTime {
|
||||
maxTime = int64(t)
|
||||
}
|
||||
}
|
||||
if t, ok := m["created_time"].(float64); ok && int64(t) > maxTime {
|
||||
maxTime = int64(t)
|
||||
}
|
||||
rows = append(rows, m)
|
||||
rows = append(rows, flat)
|
||||
}
|
||||
}
|
||||
|
||||
if pi, ok := r.Data["page_info"].(map[string]interface{}); ok {
|
||||
nextCursor := ""
|
||||
if cursorPath != "" {
|
||||
cp := strings.Split(cursorPath, ".")
|
||||
cc := respMap
|
||||
for i, p := range cp {
|
||||
if i == len(cp)-1 {
|
||||
if s, ok := cc[p].(string); ok {
|
||||
nextCursor = s
|
||||
}
|
||||
} else if m, ok := cc[p].(map[string]interface{}); ok {
|
||||
cc = m
|
||||
}
|
||||
}
|
||||
}
|
||||
if pi, ok := dataContainer["page_info"].(map[string]interface{}); ok {
|
||||
if tp, ok := pi["total_page"].(float64); ok {
|
||||
totalPages = int(tp)
|
||||
} else if tp, ok := pi["total_page"].(int); ok {
|
||||
totalPages = tp
|
||||
}
|
||||
}
|
||||
return rows, totalPages, maxTime, nextCursor, nil
|
||||
}
|
||||
|
||||
return rows, totalPages, maxTime, nil
|
||||
// flattenRow 展平嵌套 map：将子 map 的字段递归合并到顶层
|
||||
// 数组类型的字段保持原样不展平
|
||||
func flattenRow(m map[string]interface{}) map[string]interface{} {
|
||||
result := make(map[string]interface{}, len(m))
|
||||
for k, v := range m {
|
||||
if sub, ok := v.(map[string]interface{}); ok {
|
||||
// 子 map 递归展平后合并到顶层
|
||||
for sk, sv := range flattenRow(sub) {
|
||||
result[sk] = sv
|
||||
}
|
||||
} else {
|
||||
result[k] = v
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// parseResp 兼容旧版，保持4个返回值
|
||||
func parseResp(raw []byte, responseConfig map[string]interface{}) ([]map[string]interface{}, int, int64, error) {
|
||||
rows, tp, mt, _, err := parseRespExt(raw, responseConfig)
|
||||
return rows, tp, mt, err
|
||||
}
|
||||
|
||||
func savePage(ctx context.Context, td *TableDefinition, rows []map[string]interface{}) (int, error) {
|
||||
@@ -631,18 +926,17 @@ func updateSyncTime(ctx context.Context, platformCode, interfaceCode string, t i
|
||||
Save()
|
||||
}
|
||||
|
||||
func recordFailure(ctx context.Context, platformCode, interfaceCode, errMsg string) {
|
||||
func recordFailure(ctx context.Context, platformCode, interfaceCode, taskType, errMsg string) {
|
||||
dao.SyncTaskLog.Create(ctx, &taskDto.CreateSyncTaskLogReq{
|
||||
TaskID: fmt.Sprintf("%s_%s_%d", platformCode, interfaceCode, time.Now().UnixNano()),
|
||||
TaskType: fmt.Sprintf("%s_%s", platformCode, interfaceCode),
|
||||
TaskType: taskType,
|
||||
PlatformCode: platformCode,
|
||||
InterfaceCode: interfaceCode,
|
||||
Status: "failed",
|
||||
MaxRetry: 3,
|
||||
StartTime: time.Now(),
|
||||
RequestParams: map[string]interface{}{
|
||||
"platform_code": platformCode,
|
||||
"interface_code": interfaceCode,
|
||||
"error": errMsg,
|
||||
"error": errMsg,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -15,6 +15,8 @@ import (
|
||||
type PlatformConfig struct {
|
||||
*entity.DatasourcePlatform
|
||||
AccessToken string
|
||||
AppKey string
|
||||
AppSecret string
|
||||
}
|
||||
|
||||
// GetApiUrl 拼接完整 API URL
|
||||
@@ -52,6 +54,15 @@ func (m *PlatformManager) GetPlatform(ctx context.Context, platformCode string)
|
||||
}
|
||||
case "API_KEY":
|
||||
cfg.AccessToken = platform.ApiKey
|
||||
case "SIGN":
|
||||
if platform.AuthConfig != nil {
|
||||
if ak, _ := platform.AuthConfig["app_key"].(string); ak != "" {
|
||||
cfg.AppKey = ak
|
||||
}
|
||||
if as, _ := platform.AuthConfig["app_secret"].(string); as != "" {
|
||||
cfg.AppSecret = as
|
||||
}
|
||||
}
|
||||
default:
|
||||
logrus.Warnf("平台 %s 认证类型 %s 未处理", platformCode, platform.AuthType)
|
||||
}
|
||||
|
||||
@@ -12,26 +12,15 @@ import (
|
||||
"github.com/sirupsen/logrus"
|
||||
)
|
||||
|
||||
// StartAutoSync 启动自动同步（独立 goroutine，启动后自动循环执行）
|
||||
// StartAutoSync 启动自动同步（独立 goroutine，每次完成后等待 interval 再执行下一次）
|
||||
func StartAutoSync(ctx context.Context) {
|
||||
interval := GetSyncInterval(ctx)
|
||||
logrus.Infof("自动同步调度器启动，间隔: %d 分钟", interval)
|
||||
|
||||
// 首次执行：根据 sync_tracker 是否有记录自动判断全量/增量
|
||||
// 无记录 → 全量，有记录 → 增量
|
||||
runAutoSync(ctx)
|
||||
|
||||
ticker := time.NewTicker(time.Duration(interval) * time.Minute)
|
||||
defer ticker.Stop()
|
||||
logrus.Infof("自动同步调度器启动，间隔: %d 分钟（完成一次后开始计时）", interval)
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ticker.C:
|
||||
runAutoSync(ctx)
|
||||
case <-ctx.Done():
|
||||
logrus.Info("自动同步调度器已停止")
|
||||
return
|
||||
}
|
||||
runAutoSync(ctx)
|
||||
logrus.Infof("自动同步完成，等待 %d 分钟后执行下一次", interval)
|
||||
time.Sleep(time.Duration(interval) * time.Minute)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -86,18 +86,29 @@ func buildCreateSQL(td *TableDefinition) string {
|
||||
}
|
||||
cols = append(cols, "raw_data JSONB DEFAULT '{}'")
|
||||
|
||||
// 添加复合唯一索引（用于 ON CONFLICT upsert，所有 conflict_keys 作为一个复合索引）
|
||||
// 添加复合唯一索引（用于 ON CONFLICT upsert）
|
||||
var constraints []string
|
||||
if len(td.ConflictKeys) > 0 {
|
||||
cols := strings.Join(td.ConflictKeys, ", ")
|
||||
ck := strings.Join(td.ConflictKeys, ", ")
|
||||
indexName := fmt.Sprintf("uq_%s_conflict", td.TableName)
|
||||
constraints = append(constraints,
|
||||
fmt.Sprintf("CREATE UNIQUE INDEX IF NOT EXISTS %s ON %s (%s)", indexName, td.TableName, cols))
|
||||
fmt.Sprintf("CREATE UNIQUE INDEX IF NOT EXISTS %s ON %s (%s)", indexName, td.TableName, ck))
|
||||
}
|
||||
|
||||
sql := fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s (\n %s\n);\n", td.TableName, strings.Join(cols, ",\n "))
|
||||
|
||||
// 添加唯一索引
|
||||
if len(constraints) > 0 {
|
||||
sql += strings.Join(constraints, ";\n") + ";"
|
||||
sql += strings.Join(constraints, ";\n") + ";\n"
|
||||
}
|
||||
|
||||
// 添加字段注释（COMMENT ON COLUMN）
|
||||
for _, c := range td.Columns {
|
||||
if c.Comment != "" {
|
||||
escaped := strings.ReplaceAll(c.Comment, "'", "''")
|
||||
sql += fmt.Sprintf("COMMENT ON COLUMN %s.%s IS '%s';\n", td.TableName, c.Name, escaped)
|
||||
}
|
||||
}
|
||||
|
||||
return sql
|
||||
}
|
||||
|
||||
@@ -40,13 +40,15 @@ INSERT INTO api_datasource_platform (
|
||||
-- =============================================
|
||||
|
||||
-- 2.1 账户关系接口（先获取所有账户ID）
|
||||
-- 出参：{"code":0, "message":"ok", "data":{"list":[...], "page_info":{...}}}
|
||||
-- 列表路径：data.list，分页：data.page_info.total_page
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, table_definition
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
1,
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = 'tencent'),
|
||||
'账户列表', 'account_relation',
|
||||
'/advertiser/get', 'GET', 'active', 'inherit',
|
||||
'{
|
||||
@@ -58,6 +60,12 @@ INSERT INTO api_interface (
|
||||
"pagination_mode": "PAGINATION_MODE_NORMAL",
|
||||
"fields": ["account_id", "corporation_name", "is_adx", "is_bid", "is_mp"]
|
||||
}'::jsonb,
|
||||
'{
|
||||
"success_field": "code",
|
||||
"success_value": 0,
|
||||
"message_field": "message",
|
||||
"list_path": "data.list"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "tencent_account_relation",
|
||||
"columns": [
|
||||
@@ -72,13 +80,15 @@ INSERT INTO api_interface (
|
||||
);
|
||||
|
||||
-- 2.2 图片素材接口（遍历每个账户拉取图片）
|
||||
-- 出参：{"code":0, "message":"ok", "data":{"list":[...], "page_info":{...}}}
|
||||
-- 列表路径：data.list，分页：data.page_info.total_page
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, table_definition
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
1,
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = 'tencent'),
|
||||
'图片素材', 'image',
|
||||
'/images/get', 'GET', 'active', 'inherit',
|
||||
'{
|
||||
@@ -96,6 +106,12 @@ INSERT INTO api_interface (
|
||||
"value_field": "account_id"
|
||||
}
|
||||
}'::jsonb,
|
||||
'{
|
||||
"success_field": "code",
|
||||
"success_value": 0,
|
||||
"message_field": "message",
|
||||
"list_path": "data.list"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "tencent_image",
|
||||
"columns": [
|
||||
@@ -120,7 +136,7 @@ INSERT INTO api_interface (
|
||||
{"name": "quality_status", "type": "VARCHAR(50)", "comment": "质量状态"},
|
||||
{"name": "aigc_flag", "type": "VARCHAR(50)", "comment": "AIGC标志"},
|
||||
{"name": "aigc_type", "type": "INT", "comment": "AIGC类型"},
|
||||
{"name": "verify_status", "type": "VARCHAR(50) DEFAULT 'PENDING'", "comment": "校验状态"},
|
||||
{"name": "verify_status", "type": "VARCHAR(50)", "comment": "校验状态"},
|
||||
{"name": "verified_at", "type": "TIMESTAMP WITH TIME ZONE", "comment": "校验时间"},
|
||||
{"name": "verified_by", "type": "VARCHAR(64)", "comment": "校验人"}
|
||||
],
|
||||
@@ -129,13 +145,15 @@ INSERT INTO api_interface (
|
||||
);
|
||||
|
||||
-- 2.3 视频素材接口（遍历每个账户拉取视频）
|
||||
-- 出参：{"code":0, "message":"ok", "data":{"list":[...], "page_info":{...}}}
|
||||
-- 列表路径：data.list，分页：data.page_info.total_page
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, table_definition
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
1,
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = 'tencent'),
|
||||
'视频素材', 'video',
|
||||
'/videos/get', 'GET', 'active', 'inherit',
|
||||
'{
|
||||
@@ -153,6 +171,12 @@ INSERT INTO api_interface (
|
||||
"value_field": "account_id"
|
||||
}
|
||||
}'::jsonb,
|
||||
'{
|
||||
"success_field": "code",
|
||||
"success_value": 0,
|
||||
"message_field": "message",
|
||||
"list_path": "data.list"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "tencent_video",
|
||||
"columns": [
|
||||
@@ -182,7 +206,7 @@ INSERT INTO api_interface (
|
||||
{"name": "quality_status", "type": "VARCHAR(50)", "comment": "质量状态"},
|
||||
{"name": "aigc_flag", "type": "VARCHAR(50)", "comment": "AIGC标志"},
|
||||
{"name": "muse_aigc_version", "type": "INT", "comment": "Muse AIGC版本"},
|
||||
{"name": "verify_status", "type": "VARCHAR(50) DEFAULT 'PENDING'", "comment": "校验状态"},
|
||||
{"name": "verify_status", "type": "VARCHAR(50)", "comment": "校验状态"},
|
||||
{"name": "verified_at", "type": "TIMESTAMP WITH TIME ZONE", "comment": "校验时间"},
|
||||
{"name": "verified_by", "type": "VARCHAR(64)", "comment": "校验人"}
|
||||
],
|
||||
@@ -192,13 +216,15 @@ INSERT INTO api_interface (
|
||||
|
||||
-- 2.4 音频素材接口（POST + JSON Body，无需遍历账户）
|
||||
-- 注意：此接口不依赖 account_id，不依赖 prefetch，不支持增量
|
||||
-- 出参：{"code":0, "message":"ok", "data":{"list":[...], "page_info":{...}}}
|
||||
-- 列表路径：data.list，分页：data.page_info.total_page
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, table_definition
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
1,
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = 'tencent'),
|
||||
'音频素材', 'audio',
|
||||
'/muse_audios/get', 'POST', 'active', 'inherit',
|
||||
'{
|
||||
@@ -208,6 +234,12 @@ INSERT INTO api_interface (
|
||||
"page_size_param": "page_size",
|
||||
"fields": ["audio_id", "cover_image_url", "audio_name", "author", "duration", "expire_time", "feel_tags", "genre_tags"]
|
||||
}'::jsonb,
|
||||
'{
|
||||
"success_field": "code",
|
||||
"success_value": 0,
|
||||
"message_field": "message",
|
||||
"list_path": "data.list"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "tencent_audio",
|
||||
"columns": [
|
||||
@@ -219,7 +251,7 @@ INSERT INTO api_interface (
|
||||
{"name": "expire_time", "type": "BIGINT", "comment": "过期时间戳"},
|
||||
{"name": "feel_tags", "type": "JSONB", "comment": "情感标签"},
|
||||
{"name": "genre_tags", "type": "JSONB", "comment": "风格标签"},
|
||||
{"name": "verify_status", "type": "VARCHAR(50) DEFAULT 'PENDING'", "comment": "校验状态"},
|
||||
{"name": "verify_status", "type": "VARCHAR(50)", "comment": "校验状态"},
|
||||
{"name": "verified_at", "type": "TIMESTAMP WITH TIME ZONE", "comment": "校验时间"},
|
||||
{"name": "verified_by", "type": "VARCHAR(64)", "comment": "校验人"}
|
||||
],
|
||||
|
||||
60
sql/seed_data_kuaishou.sql
Normal file
60
sql/seed_data_kuaishou.sql
Normal file
@@ -0,0 +1,60 @@
|
||||
-- =============================================
|
||||
-- 快手电商平台初始化数据（18个接口）
|
||||
-- =============================================
|
||||
|
||||
INSERT INTO api_datasource_platform (tenant_id,creator,created_at,updater,updated_at,platform_code,platform_name,description,status,api_base_url,auth_type,auth_config,rate_limit_per_minute,rate_limit_per_hour,concurrency_limit,request_timeout_ms,max_retries,retry_delay_ms) VALUES
|
||||
(1,'admin',NOW(),'admin',NOW(),'kuaishou','快手电商','快手电商开放平台数据同步','ACTIVE','https://openapi.kwaixiaodian.com','API_KEY','{"sign_algorithm":"md5_upper","app_key":"YOUR_APP_KEY","app_secret":"YOUR_APP_SECRET","token_in_query":true,"query_key":"access_token","extra_query_params":{"timestamp":"{timestamp_ms}","signMethod":"MD5"}}'::jsonb,100,3600,5,30000,3,1000);
|
||||
|
||||
-- 2. 订单列表（游标分页，增量range）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'订单列表','order_list','/open/order/cursor/list','GET','active','inherit','{"parameters_location":"query","page_param":"cursor","page_size_param":"pageSize","cursor_pagination":true,"method":"open.order.cursor.list","version":1,"signMethod":"MD5","orderViewStatus":1,"pageSize":50,"sort":1,"queryType":2,"time_field":"updateTime","time_field_mode":"range","cpsType":0,"body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"]}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"data.orderList","cursor_field":"data.cursor","cursor_end_marker":"nomore"}'::jsonb,'{"table_name":"kuaishou_order_list","columns":[{"name":"oid","type":"BIGINT","comment":"订单ID"},{"name":"status","type":"INT","comment":"订单状态码"},{"name":"createTime","type":"BIGINT","comment":"创建时间"},{"name":"updateTime","type":"BIGINT","comment":"更新时间"},{"name":"payTime","type":"BIGINT","comment":"支付时间"},{"name":"sendTime","type":"BIGINT","comment":"发货时间"},{"name":"recvTime","type":"BIGINT","comment":"收货时间"},{"name":"refundTime","type":"BIGINT","comment":"退款时间"},{"name":"totalFee","type":"BIGINT","comment":"总金额(分)"},{"name":"expressFee","type":"BIGINT","comment":"运费(分)"},{"name":"discountFee","type":"BIGINT","comment":"优惠金额(分)"},{"name":"originalPrice","type":"BIGINT","comment":"原价(分)"},{"name":"buyerOpenId","type":"VARCHAR(100)","comment":"买家ID"},{"name":"sellerOpenId","type":"VARCHAR(100)","comment":"卖家ID"},{"name":"buyerNick","type":"VARCHAR(100)","comment":"买家昵称"},{"name":"sellerNick","type":"VARCHAR(100)","comment":"卖家昵称"},{"name":"remark","type":"TEXT","comment":"买家留言"},{"name":"itemTitle","type":"VARCHAR(300)","comment":"商品标题"},{"name":"num","type":"INT","comment":"商品数量"},{"name":"price","type":"BIGINT","comment":"商品单价(分)"},{"name":"activityType","type":"INT","comment":"活动类型"},{"name":"cpsType","type":"INT","comment":"分销类型"},{"name":"payType","type":"INT","comment":"支付类型"},{"name":"payChannel","type":"VARCHAR(50)","comment":"支付渠道"},{"name":"channel","type":"VARCHAR(50)","comment":"分销渠道"},{"name":"commentStatus","type":"INT","comment":"评价状态"},{"name":"priorityDelivery","type":"BOOLEAN","comment":"是否优先发货"},{"name":"carrierType","type":"INT","comment":"承运商类型"},{"name":"carrierId","type":"BIGINT","comment":"承运商ID"},{"name":"province","type":"VARCHAR(100)","comment":"省份"},{"name":"city","type":"VARCHAR(100)","comment":"城市"},{"name":"district","type":"VARCHAR(100)","comment":"区县"},{"name":"provinceCode","type":"INT","comment":"省份编码"},{"name":"cityCode","type":"INT","comment":"城市编码"},{"name":"districtCode","type":"INT","comment":"区县编码"}],"conflict_keys":["oid"]}'::jsonb);
|
||||
|
||||
-- 3. 订单详情（prefetch→order_list）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'订单详情','order_detail','/open/order/detail','GET','active','inherit','{"parameters_location":"query","pageSize":10,"page_param":"cursor","page_size_param":"pageSize","method":"open.order.detail","version":1,"signMethod":"MD5","body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"],"prefetch":{"url":"/open/order/cursor/list","method":"GET","response_path":"data.orderList","target_param":"oid","value_field":"oid"}}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"data","single_record":true}'::jsonb,'{"table_name":"kuaishou_order_detail","columns":[{"name":"oid","type":"BIGINT","comment":"订单ID"},{"name":"status","type":"INT","comment":"订单状态码"},{"name":"createTime","type":"BIGINT","comment":"创建时间"},{"name":"updateTime","type":"BIGINT","comment":"更新时间"},{"name":"payTime","type":"BIGINT","comment":"支付时间"},{"name":"sendTime","type":"BIGINT","comment":"发货时间"},{"name":"recvTime","type":"BIGINT","comment":"收货时间"},{"name":"refundTime","type":"BIGINT","comment":"退款时间"},{"name":"totalFee","type":"BIGINT","comment":"总金额(分)"},{"name":"expressFee","type":"BIGINT","comment":"运费(分)"},{"name":"discountFee","type":"BIGINT","comment":"优惠金额(分)"},{"name":"originalPrice","type":"BIGINT","comment":"原价(分)"},{"name":"buyerOpenId","type":"VARCHAR(100)","comment":"买家ID"},{"name":"sellerOpenId","type":"VARCHAR(100)","comment":"卖家ID"},{"name":"buyerNick","type":"VARCHAR(100)","comment":"买家昵称"},{"name":"sellerNick","type":"VARCHAR(100)","comment":"卖家昵称"},{"name":"remark","type":"TEXT","comment":"买家留言"},{"name":"itemTitle","type":"VARCHAR(300)","comment":"商品标题"},{"name":"num","type":"INT","comment":"商品数量"},{"name":"price","type":"BIGINT","comment":"商品单价(分)"},{"name":"activityType","type":"INT","comment":"活动类型"},{"name":"cpsType","type":"INT","comment":"分销类型"},{"name":"payType","type":"INT","comment":"支付类型"},{"name":"payChannel","type":"VARCHAR(50)","comment":"支付渠道"},{"name":"channel","type":"VARCHAR(50)","comment":"分销渠道"},{"name":"commentStatus","type":"INT","comment":"评价状态"},{"name":"priorityDelivery","type":"BOOLEAN","comment":"是否优先发货"},{"name":"carrierType","type":"INT","comment":"承运商类型"},{"name":"carrierId","type":"BIGINT","comment":"承运商ID"},{"name":"province","type":"VARCHAR(100)","comment":"省份"},{"name":"city","type":"VARCHAR(100)","comment":"城市"},{"name":"district","type":"VARCHAR(100)","comment":"区县"},{"name":"provinceCode","type":"INT","comment":"省份编码"},{"name":"cityCode","type":"INT","comment":"城市编码"},{"name":"districtCode","type":"INT","comment":"区县编码"}],"conflict_keys":["oid"]}'::jsonb);
|
||||
|
||||
-- 4. 商品列表（普通分页，全量）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'商品列表','item_list','/open/item/list/get','GET','active','inherit','{"parameters_location":"query","pageSize":20,"page_param":"pageNumber","page_size_param":"pageSize","method":"open.item.list.get","version":1,"signMethod":"MD5","itemStatus":1,"itemType":1,"onOfflineStatus":1,"body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"]}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"data.items"}'::jsonb,'{"table_name":"kuaishou_item_list","columns":[{"name":"kwaiItemId","type":"BIGINT","comment":"商品ID"},{"name":"relItemId","type":"BIGINT","comment":"外部商品ID"},{"name":"title","type":"VARCHAR(300)","comment":"商品标题"},{"name":"details","type":"TEXT","comment":"商品详情描述"},{"name":"categoryId","type":"INT","comment":"类目ID"},{"name":"categoryName","type":"VARCHAR(200)","comment":"类目名称"},{"name":"price","type":"BIGINT","comment":"商品价格"},{"name":"volume","type":"INT","comment":"销量"},{"name":"status","type":"INT","comment":"商品状态"},{"name":"auditStatus","type":"INT","comment":"审核状态"},{"name":"auditReason","type":"VARCHAR(500)","comment":"审核原因"},{"name":"shelfStatus","type":"INT","comment":"上下架状态"},{"name":"itemType","type":"INT","comment":"商品类型"},{"name":"createTime","type":"BIGINT","comment":"创建时间"},{"name":"updateTime","type":"BIGINT","comment":"更新时间"},{"name":"expressTemplateId","type":"BIGINT","comment":"运费模板ID"},{"name":"linkUrl","type":"TEXT","comment":"商品链接"}],"conflict_keys":["kwaiItemId"]}'::jsonb);
|
||||
|
||||
-- 5. 商品详情（prefetch→item_list）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'商品详情','item_detail','/open/item/get','GET','active','inherit','{"parameters_location":"query","pageSize":10,"page_param":"cursor","page_size_param":"pageSize","method":"open.item.get","version":1,"signMethod":"MD5","body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"],"prefetch":{"url":"/open/item/list/get","method":"GET","response_path":"data.items","target_param":"kwaiItemId","value_field":"kwaiItemId"}}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"data","single_record":true}'::jsonb,'{"table_name":"kuaishou_item_detail","columns":[{"name":"itemId","type":"BIGINT","comment":"商品ID"},{"name":"relItemId","type":"BIGINT","comment":"关联商品ID"},{"name":"title","type":"VARCHAR(300)","comment":"商品标题"},{"name":"details","type":"TEXT","comment":"商品详情描述"},{"name":"categoryId","type":"INT","comment":"类目ID"},{"name":"categoryName","type":"VARCHAR(200)","comment":"类目名称"},{"name":"parentCategoryId","type":"INT","comment":"父类目ID"},{"name":"parentCategoryName","type":"VARCHAR(200)","comment":"父类目名称"},{"name":"rootCategoryId","type":"INT","comment":"根类目ID"},{"name":"rootCategoryName","type":"VARCHAR(200)","comment":"根类目名称"},{"name":"auditStatus","type":"INT","comment":"审核状态 2通过"},{"name":"auditReason","type":"VARCHAR(500)","comment":"审核原因"},{"name":"onOfflineStatus","type":"INT","comment":"上下架状态 1上架 0下架"},{"name":"createdTime","type":"BIGINT","comment":"创建时间"},{"name":"updateTime","type":"BIGINT","comment":"更新时间"},{"name":"expressTemplateId","type":"BIGINT","comment":"运费模板ID"},{"name":"linkUrl","type":"TEXT","comment":"商品链接"},{"name":"purchaseLimit","type":"BOOLEAN","comment":"是否限购"},{"name":"limitCount","type":"INT","comment":"限购数量"},{"name":"timeOfSale","type":"BIGINT","comment":"开售时间"},{"name":"itemRemark","type":"TEXT","comment":"商品备注"},{"name":"spuId","type":"BIGINT","comment":"SPU ID"},{"name":"shortTitle","type":"VARCHAR(300)","comment":"商品短标题"},{"name":"sellingPoint","type":"TEXT","comment":"商品卖点"},{"name":"instructions","type":"TEXT","comment":"使用说明"},{"name":"duplicationStatus","type":"INT","comment":"铺货状态"},{"name":"duplicationReason","type":"VARCHAR(500)","comment":"铺货原因"},{"name":"multipleStock","type":"BOOLEAN","comment":"是否多库存"},{"name":"contractPhone","type":"BOOLEAN","comment":"是否合约机"},{"name":"offlineReason","type":"TEXT","comment":"下架原因"},{"name":"whiteBaseImageUrl","type":"TEXT","comment":"白底图URL"},{"name":"transparentImageUrl","type":"TEXT","comment":"透明图URL"}],"conflict_keys":["itemId"]}'::jsonb);
|
||||
|
||||
-- 6. SKU列表（prefetch→item_list）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'SKU列表','sku_list','/open/item/sku/list/get','GET','active','inherit','{"parameters_location":"query","pageSize":100,"page_param":"cursor","page_size_param":"pageSize","method":"open.item.sku.list.get","version":1,"signMethod":"MD5","skuStatus":1,"body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"],"prefetch":{"url":"/open/item/list/get","method":"GET","response_path":"data.items","target_param":"kwaiItemId","value_field":"kwaiItemId"}}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"data.skuList"}'::jsonb,'{"table_name":"kuaishou_sku_list","columns":[{"name":"kwaiSkuId","type":"BIGINT","comment":"SKU ID"},{"name":"relSkuId","type":"BIGINT","comment":"外部SKU ID"},{"name":"kwaiItemId","type":"BIGINT","comment":"商品ID"},{"name":"skuStock","type":"INT","comment":"库存"},{"name":"imageUrl","type":"TEXT","comment":"SKU图片URL"},{"name":"skuSalePrice","type":"BIGINT","comment":"售价(分)"},{"name":"volume","type":"INT","comment":"销量"},{"name":"isValid","type":"INT","comment":"是否有效 1有效"},{"name":"createTime","type":"BIGINT","comment":"创建时间"},{"name":"updateTime","type":"BIGINT","comment":"更新时间"},{"name":"specification","type":"VARCHAR(200)","comment":"规格描述"},{"name":"appkey","type":"VARCHAR(100)","comment":"应用标识"},{"name":"skuNick","type":"VARCHAR(100)","comment":"SKU别名"},{"name":"gtinCode","type":"VARCHAR(100)","comment":"商品条形码"}],"conflict_keys":["kwaiSkuId","kwaiItemId"]}'::jsonb);
|
||||
|
||||
-- 7. 售后单列表（游标pcursor，增量range，最大1天）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'售后单列表','refund_list','/open/seller/order/refund/pcursor/list','GET','active','inherit','{"parameters_location":"query","pageSize":50,"page_param":"pcursor","page_size_param":"pageSize","cursor_pagination":true,"method":"open.seller.order.refund.pcursor.list","version":1,"signMethod":"MD5","type":9,"sort":1,"queryType":2,"currentPage":1,"body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"],"time_field":"updateTime","time_field_mode":"range"}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"data.refundOrderInfoList","cursor_field":"data.pcursor","cursor_end_marker":"nomore"}'::jsonb,'{"table_name":"kuaishou_refund_list","columns":[{"name":"refundId","type":"BIGINT","comment":"退款单ID"},{"name":"oid","type":"BIGINT","comment":"订单ID"},{"name":"itemId","type":"BIGINT","comment":"商品ID"},{"name":"skuId","type":"BIGINT","comment":"SKU ID"},{"name":"relSkuId","type":"BIGINT","comment":"外部SKU ID"},{"name":"skuNick","type":"VARCHAR(100)","comment":"SKU别名"},{"name":"handlingWay","type":"INT","comment":"售后方式"},{"name":"negotiateStatus","type":"INT","comment":"协商状态"},{"name":"refundFee","type":"BIGINT","comment":"退款金额(分)"},{"name":"refundReason","type":"INT","comment":"退款原因码"},{"name":"refundReasonDesc","type":"VARCHAR(500)","comment":"退款原因描述"},{"name":"refundDesc","type":"TEXT","comment":"退款描述"},{"name":"refundType","type":"INT","comment":"退款类型"},{"name":"status","type":"INT","comment":"退款状态"},{"name":"receiptStatus","type":"INT","comment":"收货状态"},{"name":"buyerId","type":"BIGINT","comment":"买家ID"},{"name":"sellerId","type":"BIGINT","comment":"卖家ID"},{"name":"logisticsId","type":"BIGINT","comment":"物流ID"},{"name":"relItemId","type":"BIGINT","comment":"关联商品ID"},{"name":"submitTime","type":"BIGINT","comment":"提交时间"},{"name":"createTime","type":"BIGINT","comment":"创建时间"},{"name":"updateTime","type":"BIGINT","comment":"更新时间"},{"name":"negotiateUpdateTime","type":"BIGINT","comment":"协商更新时间"},{"name":"endTime","type":"BIGINT","comment":"结束时间"},{"name":"expireTime","type":"BIGINT","comment":"过期时间"}],"conflict_keys":["refundId"]}'::jsonb);
|
||||
|
||||
-- 8. 售后单详情（prefetch→refund_list）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'售后单详情','refund_detail','/open/seller/order/refund/detail','GET','active','inherit','{"parameters_location":"query","pageSize":10,"page_param":"cursor","page_size_param":"pageSize","method":"open.seller.order.refund.detail","version":1,"signMethod":"MD5","body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"],"prefetch":{"url":"/open/seller/order/refund/pcursor/list","method":"GET","response_path":"data.refundOrderInfoList","target_param":"refundId","value_field":"refundId"}}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"data","single_record":true}'::jsonb,'{"table_name":"kuaishou_refund_detail","columns":[{"name":"refundId","type":"BIGINT","comment":"退款单ID"},{"name":"oid","type":"BIGINT","comment":"订单ID"},{"name":"itemId","type":"BIGINT","comment":"商品ID"},{"name":"skuId","type":"BIGINT","comment":"SKU ID"},{"name":"relSkuId","type":"BIGINT","comment":"外部SKU ID"},{"name":"relItemId","type":"BIGINT","comment":"关联商品ID"},{"name":"handlingWay","type":"INT","comment":"售后方式"},{"name":"negotiateStatus","type":"INT","comment":"协商状态"},{"name":"refundFee","type":"BIGINT","comment":"退款金额(分)"},{"name":"refundReason","type":"INT","comment":"退款原因码"},{"name":"refundReasonDesc","type":"VARCHAR(500)","comment":"退款原因描述"},{"name":"refundDesc","type":"TEXT","comment":"退款描述"},{"name":"refundType","type":"INT","comment":"退款类型"},{"name":"status","type":"INT","comment":"退款状态"},{"name":"receiptStatus","type":"INT","comment":"收货状态"},{"name":"buyerId","type":"BIGINT","comment":"买家ID"},{"name":"buyerOpenId","type":"VARCHAR(100)","comment":"买家openId"},{"name":"sellerId","type":"BIGINT","comment":"卖家ID"},{"name":"logisticsId","type":"BIGINT","comment":"物流ID"},{"name":"productNum","type":"INT","comment":"商品数量"},{"name":"specialRefundType","type":"INT","comment":"特殊退款类型"},{"name":"sellerDisagreeReason","type":"INT","comment":"卖家拒绝原因码"},{"name":"sellerDisagreeDesc","type":"TEXT","comment":"卖家拒绝描述"},{"name":"negotiateReason","type":"VARCHAR(500)","comment":"协商原因"},{"name":"address","type":"TEXT","comment":"收货地址"},{"name":"validNegotiateBuyerModifyTimeStamp","type":"BIGINT","comment":"有效协商修改时间戳"},{"name":"timeLimitNegotiateChange","type":"INT","comment":"协商变更时限"},{"name":"submitTime","type":"BIGINT","comment":"提交时间"},{"name":"createTime","type":"BIGINT","comment":"创建时间"},{"name":"updateTime","type":"BIGINT","comment":"更新时间"},{"name":"negotiateUpdateTime","type":"BIGINT","comment":"协商更新时间"},{"name":"endTime","type":"BIGINT","comment":"结束时间"},{"name":"expireTime","type":"BIGINT","comment":"过期时间"}],"conflict_keys":["refundId"]}'::jsonb);
|
||||
|
||||
-- 9. 商品类目（全量，data直接是数组）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'商品类目','category','/open/item/category','GET','active','inherit','{"parameters_location":"query","pageSize":100,"page_param":"cursor","page_size_param":"pageSize","method":"open.item.category","version":1,"signMethod":"MD5","body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"]}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"data"}'::jsonb,'{"table_name":"kuaishou_category","columns":[{"name":"categoryId","type":"BIGINT","comment":"类目ID"},{"name":"categoryName","type":"VARCHAR(300)","comment":"类目名称"},{"name":"categoryPid","type":"BIGINT","comment":"父级类目ID 0为根类目"}],"conflict_keys":["categoryId"]}'::jsonb);
|
||||
|
||||
-- 10. 带货口碑分（单记录，全量）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'带货口碑分','score_master','/open/score/master/get','GET','active','inherit','{"parameters_location":"query","pageSize":10,"page_param":"cursor","page_size_param":"pageSize","method":"open.score.master.get","version":1,"signMethod":"MD5","body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"]}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"data","single_record":true}'::jsonb,'{"table_name":"kuaishou_score_master","columns":[{"name":"hasData","type":"BOOLEAN","comment":"是否有数据"},{"name":"showName","type":"VARCHAR(100)","comment":"分值文案描述"},{"name":"scoreStr","type":"VARCHAR(20)","comment":"分值(保留两位小数)"}],"conflict_keys":["showName"]}'::jsonb);
|
||||
|
||||
-- 11. 店铺体验分（单记录，全量）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'店铺体验分','score_shop','/open/score/shop/get','GET','active','inherit','{"parameters_location":"query","pageSize":10,"page_param":"cursor","page_size_param":"pageSize","method":"open.score.shop.get","version":1,"signMethod":"MD5","body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"]}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"data","single_record":true}'::jsonb,'{"table_name":"kuaishou_score_shop","columns":[{"name":"hasData","type":"BOOLEAN","comment":"是否有数据"},{"name":"showName","type":"VARCHAR(100)","comment":"分值文案描述"},{"name":"scoreStr","type":"VARCHAR(20)","comment":"分值(保留两位小数)"}],"conflict_keys":["showName"]}'::jsonb);
|
||||
|
||||
-- 12. 店铺信息（单记录，全量）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'店铺信息','shop_info','/open/shop/info/get','GET','active','inherit','{"parameters_location":"query","pageSize":10,"page_param":"cursor","page_size_param":"pageSize","method":"open.shop.info.get","version":1,"signMethod":"MD5","body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"]}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"data","single_record":true}'::jsonb,'{"table_name":"kuaishou_shop_info","columns":[{"name":"shopName","type":"VARCHAR(300)","comment":"店铺全名"},{"name":"shopType","type":"INT","comment":"店铺类型 1旗舰 2专卖 3专营"},{"name":"shopExpScoreStr","type":"VARCHAR(20)","comment":"购物体验分"},{"name":"productQualityScoreStr","type":"VARCHAR(20)","comment":"商品品质分"},{"name":"contentQualifyScoreStr","type":"VARCHAR(20)","comment":"内容质量分"},{"name":"customerServiceScoreStr","type":"VARCHAR(20)","comment":"客服服务分"},{"name":"logisticsServiceScoreStr","type":"VARCHAR(20)","comment":"物流服务分"},{"name":"afterSalesServiceScoreStr","type":"VARCHAR(20)","comment":"售后服务分"}],"conflict_keys":["shopName"]}'::jsonb);
|
||||
|
||||
-- 13. 门店POI详情（单记录，需outerPoiId+source）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'门店POI详情','poi_detail','/open/shop/poi/getPoiDetailByOuterPoi','GET','active','inherit','{"parameters_location":"query","pageSize":10,"page_param":"cursor","page_size_param":"pageSize","method":"open.shop.poi.getPoiDetailByOuterPoi","version":1,"signMethod":"MD5","body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"]}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"data","single_record":true}'::jsonb,'{"table_name":"kuaishou_poi_detail","columns":[{"name":"outerPoiId","type":"VARCHAR(100)","comment":"图商poiId"},{"name":"source","type":"INT","comment":"图商来源 1高德 2百度 3腾讯"},{"name":"poiId","type":"VARCHAR(100)","comment":"快手poiId"},{"name":"name","type":"VARCHAR(300)","comment":"poi名称"},{"name":"longitude","type":"VARCHAR(50)","comment":"经度"},{"name":"latitude","type":"VARCHAR(50)","comment":"纬度"},{"name":"address","type":"TEXT","comment":"地址信息"},{"name":"country","type":"VARCHAR(100)","comment":"国家"},{"name":"province","type":"VARCHAR(100)","comment":"省份"},{"name":"city","type":"VARCHAR(100)","comment":"城市"},{"name":"district","type":"VARCHAR(100)","comment":"区县"},{"name":"provinceCode","type":"BIGINT","comment":"省份编码"},{"name":"cityCode","type":"BIGINT","comment":"城市编码"},{"name":"districtCode","type":"BIGINT","comment":"区县编码"}],"conflict_keys":["poiId"]}'::jsonb);
|
||||
|
||||
-- 14. 分销订单列表（游标pcursor，增量range，最大3天）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'分销订单列表','cps_order_list','/open/seller/order/cps/list','GET','active','inherit','{"parameters_location":"query","pageSize":80,"page_param":"pcursor","page_size_param":"pageSize","cursor_pagination":true,"method":"open.seller.order.cps.list","version":1,"signMethod":"MD5","sort":1,"queryType":2,"type":1,"currentPage":1,"body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"],"time_field":"updateTime","time_field_mode":"range"}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"data.cpsOrderList","cursor_field":"data.pcursor","cursor_end_marker":"nomore"}'::jsonb,'{"table_name":"kuaishou_cps_order_list","columns":[{"name":"oid","type":"BIGINT","comment":"订单ID"},{"name":"distributorId","type":"BIGINT","comment":"分销者ID"},{"name":"distributorName","type":"VARCHAR(200)","comment":"分销者昵称"},{"name":"sellerId","type":"BIGINT","comment":"卖家ID"},{"name":"status","type":"INT","comment":"分销单状态"},{"name":"settlementTime","type":"BIGINT","comment":"结算时间"},{"name":"settlementSuccessTime","type":"BIGINT","comment":"结算成功时间"},{"name":"refundTime","type":"BIGINT","comment":"退款时间"},{"name":"payTime","type":"BIGINT","comment":"付款时间"},{"name":"expressFee","type":"BIGINT","comment":"运费(分)"},{"name":"totalFee","type":"BIGINT","comment":"实际付款金额-运费(分)"},{"name":"commissionRate","type":"BIGINT","comment":"分销率"},{"name":"estimatedIncome","type":"BIGINT","comment":"分销金额(分)"},{"name":"createTime","type":"BIGINT","comment":"创建时间"},{"name":"updateTime","type":"BIGINT","comment":"更新时间"},{"name":"platformDpRate","type":"BIGINT","comment":"平台佣金比例 千分比"},{"name":"activityUserId","type":"BIGINT","comment":"团长ID"},{"name":"activityUserNickname","type":"VARCHAR(200)","comment":"团长昵称"},{"name":"investmentPromotionRate","type":"INT","comment":"团长服务费率 千分比"},{"name":"investmentPromotionAmount","type":"BIGINT","comment":"团长服务费金额(分)"},{"name":"buyerOpenId","type":"VARCHAR(100)","comment":"买家识别ID"},{"name":"settlementBizType","type":"INT","comment":"订单业务类型 1快分销 2聚力计划"},{"name":"promoterServiceInCome","type":"BIGINT","comment":"达人接单服务收入(分)"},{"name":"promoterExcitationInCome","type":"BIGINT","comment":"达人奖励收入(分)"},{"name":"investmentServiceInCome","type":"BIGINT","comment":"团长接单收入(分)"},{"name":"investmentExcitationInCome","type":"BIGINT","comment":"团长奖励收入(分)"},{"name":"orderChannel","type":"VARCHAR(100)","comment":"出单渠道"},{"name":"activityId","type":"BIGINT","comment":"活动ID"},{"name":"itemId","type":"BIGINT","comment":"商品ID"}],"conflict_keys":["oid"]}'::jsonb);
|
||||
|
||||
-- 15. 分销订单详情（prefetch→cps_order_list）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'分销订单详情','cps_order_detail','/open/seller/order/cps/detail','GET','active','inherit','{"parameters_location":"query","pageSize":10,"page_param":"cursor","page_size_param":"pageSize","method":"open.seller.order.cps.detail","version":1,"signMethod":"MD5","body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"],"prefetch":{"url":"/open/seller/order/cps/list","method":"GET","response_path":"data.cpsOrderList","target_param":"orderId","value_field":"oid"}}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"data","single_record":true}'::jsonb,'{"table_name":"kuaishou_cps_order_detail","columns":[{"name":"oid","type":"BIGINT","comment":"订单ID"},{"name":"distributorId","type":"BIGINT","comment":"分销者ID"},{"name":"distributorName","type":"VARCHAR(200)","comment":"分销者昵称"},{"name":"sellerId","type":"BIGINT","comment":"卖家ID"},{"name":"status","type":"INT","comment":"分销单状态"},{"name":"settlementTime","type":"BIGINT","comment":"结算时间"},{"name":"settlementSuccessTime","type":"BIGINT","comment":"结算成功时间"},{"name":"refundTime","type":"BIGINT","comment":"退款时间"},{"name":"payTime","type":"BIGINT","comment":"付款时间"},{"name":"expressFee","type":"BIGINT","comment":"运费(分)"},{"name":"totalFee","type":"BIGINT","comment":"实际付款金额-运费(分)"},{"name":"commissionRate","type":"BIGINT","comment":"分销率"},{"name":"estimatedIncome","type":"BIGINT","comment":"分销金额(分)"},{"name":"createTime","type":"BIGINT","comment":"创建时间"},{"name":"updateTime","type":"BIGINT","comment":"更新时间"},{"name":"platformDpRate","type":"BIGINT","comment":"平台佣金比例 千分比"},{"name":"activityUserId","type":"BIGINT","comment":"团长ID"},{"name":"activityUserNickname","type":"VARCHAR(200)","comment":"团长昵称"},{"name":"investmentPromotionRate","type":"INT","comment":"团长服务费率 千分比"},{"name":"investmentPromotionAmount","type":"BIGINT","comment":"团长服务费金额(分)"},{"name":"buyerOpenId","type":"VARCHAR(100)","comment":"买家识别ID"},{"name":"settlementBizType","type":"INT","comment":"订单业务类型 1快分销 2聚力计划"},{"name":"promoterServiceInCome","type":"BIGINT","comment":"达人接单服务收入(分)"},{"name":"promoterExcitationInCome","type":"BIGINT","comment":"达人奖励收入(分)"},{"name":"investmentServiceInCome","type":"BIGINT","comment":"团长接单收入(分)"},{"name":"investmentExcitationInCome","type":"BIGINT","comment":"团长奖励收入(分)"},{"name":"orderChannel","type":"VARCHAR(100)","comment":"出单渠道"},{"name":"activityId","type":"BIGINT","comment":"活动ID"},{"name":"itemId","type":"BIGINT","comment":"商品ID"}],"conflict_keys":["oid"]}'::jsonb);
|
||||
|
||||
-- 16. 达人分销订单列表（游标pcursor，增量range，最大7天）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'达人分销订单列表','distributor_order_list','/open/distribution/cps/distributor/order/cursor/list','GET','active','inherit','{"parameters_location":"query","pageSize":100,"page_param":"pcursor","page_size_param":"pageSize","cursor_pagination":true,"method":"open.distribution.cps.distributor.order.cursor.list","version":1,"signMethod":"MD5","sortType":1,"queryType":2,"cpsOrderStatus":0,"body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"],"time_field":"updateTime","time_field_mode":"range"}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"data.orderView","cursor_field":"data.pcursor","cursor_end_marker":"nomore"}'::jsonb,'{"table_name":"kuaishou_distributor_order_list","columns":[{"name":"oid","type":"BIGINT","comment":"订单ID"},{"name":"distributorId","type":"BIGINT","comment":"分销者ID"},{"name":"cpsOrderStatus","type":"INT","comment":"分销订单状态"},{"name":"orderCreateTime","type":"BIGINT","comment":"订单创建时间"},{"name":"payTime","type":"BIGINT","comment":"支付时间"},{"name":"sendTime","type":"BIGINT","comment":"发货时间"},{"name":"sendStatus","type":"INT","comment":"发货状态"},{"name":"recvTime","type":"BIGINT","comment":"收货时间"},{"name":"orderTradeAmount","type":"BIGINT","comment":"订单交易金额(分)"},{"name":"baseAmount","type":"BIGINT","comment":"结算基数"},{"name":"shareRateStr","type":"VARCHAR(20)","comment":"分佣比例"},{"name":"settlementBizType","type":"INT","comment":"订单业务类型"},{"name":"settlementAmount","type":"BIGINT","comment":"结算金额(分)"},{"name":"settlementSuccessTime","type":"BIGINT","comment":"结算成功时间"},{"name":"buyerOpenId","type":"VARCHAR(100)","comment":"买家识别ID"},{"name":"createTime","type":"BIGINT","comment":"创建时间"},{"name":"updateTime","type":"BIGINT","comment":"更新时间"}],"conflict_keys":["oid"]}'::jsonb);
|
||||
|
||||
-- 17. 开票金额查询（单记录，prefetch→order_list）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'开票金额','invoice_amount','/open/invoice/amount/get','GET','active','inherit','{"parameters_location":"query","pageSize":10,"page_param":"cursor","page_size_param":"pageSize","method":"open.invoice.amount.get","version":1,"signMethod":"MD5","fromType":"1","body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"],"prefetch":{"url":"/open/order/cursor/list","method":"GET","response_path":"data.orderList","target_param":"orderId","value_field":"oid"}}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"queryInvoiceAmountResponseData","single_record":true}'::jsonb,'{"table_name":"kuaishou_invoice_amount","columns":[{"name":"toReceiverInvoiceAmount","type":"VARCHAR(50)","comment":"达人开票金额"},{"name":"toBuyerInvoiceAmount","type":"VARCHAR(50)","comment":"买家开票金额"},{"name":"toPlatformInvoiceAmount","type":"VARCHAR(50)","comment":"平台开票金额"},{"name":"orderStatus","type":"BIGINT","comment":"订单状态"},{"name":"preSaleOrder","type":"BOOLEAN","comment":"是否定金预售"},{"name":"platformAllowanceAmount","type":"VARCHAR(50)","comment":"平台补贴"},{"name":"freightWhenOrder","type":"VARCHAR(50)","comment":"运费"},{"name":"orderId","type":"BIGINT","comment":"订单ID"},{"name":"userPayAmount","type":"VARCHAR(50)","comment":"用户实付"},{"name":"receiverSubsidyAmount","type":"VARCHAR(50)","comment":"达人补贴"},{"name":"queryTime","type":"VARCHAR(50)","comment":"查询时间"}],"conflict_keys":["orderId"]}'::jsonb);
|
||||
|
||||
-- 18. 代发订单列表（游标cursor，POST，全量最大30天）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'代发订单列表','dropshipping_order_list','/open/dropshipping/order/merchant/list','POST','active','inherit','{"parameters_location":"query","pageSize":50,"page_param":"cursor","page_size_param":"pageSize","cursor_pagination":true,"method":"open.dropshipping.order.merchant.list","version":1,"signMethod":"MD5","dropshippingStatus":0,"orderStatus":0,"refundStatus":0,"orderType":200,"queryType":"PAT_TIME","sort":0,"body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"]}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"data.orderList","cursor_field":"data.cursor","cursor_end_marker":"nomore"}'::jsonb,'{"table_name":"kuaishou_dropshipping_order_list","columns":[{"name":"oid","type":"BIGINT","comment":"交易订单ID"},{"name":"dropshippingOrderCode","type":"VARCHAR(100)","comment":"代发订单编码"},{"name":"payTime","type":"BIGINT","comment":"支付时间"},{"name":"buyerOpenId","type":"VARCHAR(100)","comment":"买家开放id"},{"name":"buyerNick","type":"VARCHAR(100)","comment":"买家昵称"},{"name":"sellerOpenId","type":"VARCHAR(100)","comment":"卖家开放id"},{"name":"sellerNick","type":"VARCHAR(100)","comment":"卖家昵称"},{"name":"orderStatus","type":"INT","comment":"订单状态"},{"name":"orderStatusDesc","type":"VARCHAR(100)","comment":"订单状态描述"},{"name":"refundStatus","type":"INT","comment":"售后状态"},{"name":"refundStatusDesc","type":"VARCHAR(100)","comment":"售后状态描述"},{"name":"orderType","type":"INT","comment":"订单类型"},{"name":"orderTypeDesc","type":"VARCHAR(100)","comment":"订单类型描述"},{"name":"deliveryTime","type":"BIGINT","comment":"发货时间"},{"name":"createTime","type":"BIGINT","comment":"创建时间"},{"name":"updateTime","type":"BIGINT","comment":"修改时间"},{"name":"waybillCode","type":"VARCHAR(100)","comment":"运单号"},{"name":"expressCompanyCode","type":"VARCHAR(50)","comment":"物流公司编码"},{"name":"expressCompanyName","type":"VARCHAR(100)","comment":"物流公司名称"},{"name":"factoryCode","type":"VARCHAR(100)","comment":"代发厂家编码"},{"name":"factoryName","type":"VARCHAR(200)","comment":"代发厂家名称"},{"name":"allocateTime","type":"BIGINT","comment":"分配时间"},{"name":"dropshippingStatus","type":"INT","comment":"代发状态"},{"name":"dropshippingStatusDesc","type":"VARCHAR(100)","comment":"代发状态描述"},{"name":"cancelAllocateTime","type":"BIGINT","comment":"取消分配时间"},{"name":"cancelAllocateReason","type":"TEXT","comment":"取消分配原因"},{"name":"name","type":"VARCHAR(100)","comment":"收货人姓名"},{"name":"mobile","type":"VARCHAR(50)","comment":"手机号码"},{"name":"provinceName","type":"VARCHAR(100)","comment":"省份"},{"name":"cityName","type":"VARCHAR(100)","comment":"城市"},{"name":"districtName","type":"VARCHAR(100)","comment":"区县"},{"name":"detailAddress","type":"TEXT","comment":"详细地址"}],"conflict_keys":["oid"]}'::jsonb);
|
||||
|
||||
-- 19. 代发订单详情（prefetch→dropshipping_order_list，单记录）
|
||||
INSERT INTO api_interface (tenant_id,creator,created_at,updater,updated_at,platform_id,name,code,url,method,status,auth_type,request_config,response_config,table_definition) VALUES (1,'admin',NOW(),'admin',NOW(),(SELECT id FROM api_datasource_platform WHERE platform_code='kuaishou'),'代发订单详情','dropshipping_order_detail','/open/dropshipping/order/merchant/detail','POST','active','inherit','{"parameters_location":"query","pageSize":10,"page_param":"cursor","page_size_param":"pageSize","method":"open.dropshipping.order.merchant.detail","version":1,"signMethod":"MD5","body_wrapper_field":"param","exclude_from_wrapper":["method","version","signMethod"],"prefetch":{"url":"/open/dropshipping/order/merchant/list","method":"POST","response_path":"data.orderList","target_param":"dropshippingOrderCode","value_field":"dropshippingOrderCode"}}'::jsonb,'{"success_field":"result","success_value":1,"list_path":"data","single_record":true}'::jsonb,'{"table_name":"kuaishou_dropshipping_order_detail","columns":[{"name":"dropshippingOrderCode","type":"VARCHAR(100)","comment":"代发订单编码"},{"name":"oid","type":"BIGINT","comment":"交易订单ID"},{"name":"payTime","type":"BIGINT","comment":"支付时间"},{"name":"buyerOpenId","type":"VARCHAR(100)","comment":"买家开放id"},{"name":"buyerNick","type":"VARCHAR(100)","comment":"买家昵称"},{"name":"sellerOpenId","type":"VARCHAR(100)","comment":"卖家开放id"},{"name":"sellerNick","type":"VARCHAR(100)","comment":"卖家昵称"},{"name":"orderStatus","type":"INT","comment":"订单状态"},{"name":"orderStatusDesc","type":"VARCHAR(100)","comment":"订单状态描述"},{"name":"refundStatus","type":"INT","comment":"售后状态"},{"name":"refundStatusDesc","type":"VARCHAR(100)","comment":"售后状态描述"},{"name":"orderType","type":"INT","comment":"订单类型"},{"name":"orderTypeDesc","type":"VARCHAR(100)","comment":"订单类型描述"},{"name":"deliveryTime","type":"BIGINT","comment":"发货时间"},{"name":"createTime","type":"BIGINT","comment":"创建时间"},{"name":"updateTime","type":"BIGINT","comment":"修改时间"},{"name":"waybillCode","type":"VARCHAR(100)","comment":"运单号"},{"name":"expressCompanyCode","type":"VARCHAR(50)","comment":"物流公司编码"},{"name":"expressCompanyName","type":"VARCHAR(100)","comment":"物流公司名称"},{"name":"factoryCode","type":"VARCHAR(100)","comment":"代发厂家编码"},{"name":"factoryName","type":"VARCHAR(200)","comment":"代发厂家名称"},{"name":"allocateTime","type":"BIGINT","comment":"分配时间"},{"name":"dropshippingStatus","type":"INT","comment":"代发状态"},{"name":"dropshippingStatusDesc","type":"VARCHAR(100)","comment":"代发状态描述"},{"name":"cancelAllocateTime","type":"BIGINT","comment":"取消分配时间"},{"name":"cancelAllocateReason","type":"TEXT","comment":"取消分配原因"},{"name":"buyerNote","type":"JSONB","comment":"买家备注列表"},{"name":"sellerNote","type":"JSONB","comment":"卖家备注列表"},{"name":"orderItemList","type":"JSONB","comment":"订单商品信息列表"},{"name":"detailAddress","type":"TEXT","comment":"详细地址"},{"name":"addressId","type":"VARCHAR(100)","comment":"地址id"},{"name":"cityCode","type":"VARCHAR(50)","comment":"城市编码"},{"name":"provinceCode","type":"VARCHAR(50)","comment":"省份编码"},{"name":"streetCode","type":"VARCHAR(50)","comment":"街道编码"},{"name":"streetName","type":"VARCHAR(100)","comment":"街道名称"},{"name":"provinceName","type":"VARCHAR(100)","comment":"省份"},{"name":"countryCode","type":"VARCHAR(50)","comment":"国家编码"},{"name":"districtName","type":"VARCHAR(100)","comment":"区县名称"},{"name":"cityName","type":"VARCHAR(100)","comment":"城市名称"},{"name":"districtCode","type":"VARCHAR(50)","comment":"区县编码"},{"name":"countryName","type":"VARCHAR(100)","comment":"国家名称"},{"name":"receiverName","type":"VARCHAR(100)","comment":"收货人姓名"},{"name":"receiverMobile","type":"VARCHAR(50)","comment":"收货人手机号码"},{"name":"receiverTelephone","type":"VARCHAR(50)","comment":"收货人电话"}],"conflict_keys":["dropshippingOrderCode"]}'::jsonb);
|
||||
Reference in New Issue
Block a user